Everyone who plays CTFs regularly has a category they scroll past. For a lot of people, it is crypto.
You open the challenge, see something like n, e, and c listed as variables, or a block of text that looks like it was produced by someone sitting on a keyboard, and you make a calculation: this will take too long, I do not have the background, I will come back to it. You never come back to it.
The problem with that calculation is that it is usually wrong. A large proportion of CTF crypto challenges do not require a mathematics degree. They require pattern recognition, a small toolkit, and the willingness to apply a systematic approach rather than panic and close the tab.
This guide is for CTF players who already know their way around other categories but have been quietly haemorrhaging points to crypto challenges for longer than they want to admit. Here is how to stop doing that.
The First Principle: Crypto Challenges Are Designed to Be Broken
Before getting into specific challenge types, this is worth saying clearly: CTF challenges are not real-world encryption. Real-world encryption, implemented correctly, is not breakable by a competitor in a weekend competition. CTF crypto challenges are designed around intentional weaknesses. The cipher is broken, the key is weak, the implementation has a flaw, the parameters were chosen badly.
Your job is not to break cryptography in general. It is to identify the specific flaw the challenge author introduced and exploit it. That is a much more tractable problem, and it starts with correctly identifying what you are looking at.
Step One: Identify Before You Solve
The single most useful habit in CTF crypto is resisting the urge to start solving before you have identified what you are dealing with. Experienced players spend the first few minutes of every crypto challenge just reading: the challenge description, the data format, any variable names, any hints in the flavour text. Challenge authors almost always signal the encryption scheme somewhere, even if it is subtle.
A few reliable recognition patterns worth memorising:
If the data is a string of letters and numbers ending in one or two equals signs, it is almost certainly Base64. If it is a long string of only hexadecimal characters (0-9, A-F), it is hex-encoded data. If every character in the ciphertext is a letter and it roughly preserves word lengths, you are probably looking at a classical cipher. If the challenge gives you variables labelled n, e, and c, you are looking at RSA. If the challenge gives you ciphertext alongside source code that shows a key being applied byte by byte, you are probably looking at XOR.
Pattern recognition like this does not require deep cryptographic knowledge. It requires exposure. The more challenges you have seen, the faster you recognise the shape of what you are dealing with.
Encoding Challenges: The Easiest Points in the Category
A significant portion of beginner and some intermediate CTF crypto challenges are not encryption at all. They are encoding. This distinction matters enormously.
Encryption requires a key to reverse. Encoding does not. Base64, hexadecimal, binary, and URL encoding are representations of data, not protections of it. Once you recognise the encoding scheme, decoding it is one operation.
Base64 is the most common. The giveaway is the alphabet: only uppercase letters, lowercase letters, digits, + and /, and the trailing = or == padding. If you see that pattern, Base64 decode it immediately. Hex-encoded data looks like a long string of characters from 0-9 and A-F with no other characters. Binary is rows of 1s and 0s. ROT13 is a Caesar cipher with a fixed shift of 13, common enough that it has become its own thing.
Some challenges layer multiple encodings on top of each other. You decode something that looks like a flag but is not, and you have to run it through another operation. Do not stop when you get readable output. Ask whether that output is actually the flag or just another layer.
CyberChef is the tool for this. It is a browser-based operations chain where you can stack transformations: Base64 decode, then hex decode, then ROT13, watching the output at each stage. The "Magic" operation will even attempt to auto-detect and apply the right transforms. For anything encoding-related, CyberChef should be the first thing you open.
Classical Ciphers: Pattern Recognition Over Mathematics
Classical ciphers are the category that feels intimidating but is actually almost entirely about pattern analysis. They appear frequently in CTFs, particularly at beginner and intermediate difficulty.
The Caesar cipher shifts each letter in the alphabet by a fixed number of positions. It has 25 possible keys. You brute force it by trying all 25 shifts and reading the output. No maths required. ROT13 is just a Caesar cipher with a shift of 13. Any online Caesar solver or CyberChef will crack it in seconds.
The Vigenere cipher is more sophisticated. It applies a series of Caesar shifts according to a repeating keyword, making it resistant to simple frequency analysis. But it has a well-known weakness: if you can determine the key length, you can split the ciphertext into groups, run frequency analysis on each group separately, and recover the key one letter at a time. Tools like dcode.fr automate this entirely.
Substitution ciphers replace each letter with a different symbol or letter, using a fixed alphabet-to-alphabet mapping. The approach here is frequency analysis: in English text, e is the most common letter, followed by t, a, o, i, n. If you see one symbol appearing far more frequently than others, it is probably e. Tools like quipqiup.com automate substitution cipher solving using statistical analysis. Give it the ciphertext and let it work.
The broader lesson from classical ciphers is that these schemes are not mathematically strong. They were designed in an era before computers. The same statistical properties that make natural language readable are the properties that make these ciphers breakable. You are not breaking encryption. You are doing pattern analysis.
XOR: The Building Block Behind Many CTF Challenges
XOR is not a cipher in the classical sense, but it underpins an enormous number of CTF crypto challenges and it is worth understanding conceptually rather than just knowing it exists.
XOR (exclusive or) is a bitwise operation with a simple property: if you XOR a value with a key, you get ciphertext. If you XOR that ciphertext with the same key again, you recover the original value. Formally: plaintext XOR key = ciphertext, and ciphertext XOR key = plaintext. That symmetry is what makes it both useful as a simple encryption scheme and exploitable as a CTF challenge.
The most common CTF attack on XOR encryption is the known-plaintext attack. CTF flags almost always start with a known prefix, usually the competition name followed by a brace: THM{, FLAG{, picoCTF{. If you know the first few characters of the plaintext, you can XOR them against the first few bytes of the ciphertext to recover the beginning of the key. If the key is short and repeating, those recovered bytes are often enough to determine the full key and decrypt everything.
Single-byte XOR is even simpler. If the entire plaintext has been XORed with a single character, there are only 256 possible keys. You try all 256 and look for the one that produces readable English. This is a brute force with a trivially small search space.
CyberChef handles basic XOR decryption well. For more complex multi-byte XOR analysis, xortool is the standard command-line option.
RSA: Why It Is Not as Scary as It Looks
RSA is where most people disengage. The variables have unfamiliar names, the maths involves prime factorisation and modular arithmetic, and the Wikipedia article reads like a textbook. But the key insight for CTF purposes is this: in the real world, RSA is secure because its parameters are chosen well. In CTF challenges, RSA is breakable because the parameters are chosen badly on purpose.
You do not need to understand RSA deeply to solve most CTF RSA challenges. You need to recognise the common weakness patterns and know which tool or technique applies.
The variables you will see in RSA challenges are n (the modulus, the product of two large primes), e (the public exponent), d (the private exponent, which you want to recover), and c (the ciphertext). The security of RSA rests on the difficulty of factoring n back into its two prime components p and q. Most CTF RSA attacks work by finding a way to factor n or otherwise recover d.
The most common weaknesses you will encounter at beginner to intermediate level are small values of n that can be factored directly, related primes where p and q are close in value (Fermat's factorisation), small public exponents combined with small messages (cube root attack when e = 3), and shared moduli across multiple ciphertext and public key pairs.
For the majority of these, RsaCtfTool will do the heavy lifting. Give it the public key and ciphertext, and it automatically attempts multiple attack strategies in sequence. It does not work on every RSA challenge, but for beginner and intermediate CTF RSA, it solves a substantial proportion of problems with a single command. When it does not work, the failure modes tell you something about what kind of challenge you are dealing with.
For challenges where n looks factorable, factordb.com maintains a database of pre-computed factorisations and is worth checking before doing anything else.
The Practical Toolkit
To summarise the tools that cover the vast majority of CTF crypto challenges:
CyberChef handles all encoding and decoding challenges and provides a visual operations chain that makes layered encoding problems easy to work through.
dcode.fr is a comprehensive cipher identification and solving resource. If you are not sure what type of cipher you are looking at, its cipher identifier will often tell you. It has implementations of almost every classical cipher.
quipqiup.com solves substitution ciphers automatically using frequency analysis. Paste in the ciphertext and let it run.
RsaCtfTool automates common RSA attacks for CTF challenges. It is a command-line Python tool that takes a public key and ciphertext and attempts multiple factorisation and attack strategies.
factordb.com is a database of known integer factorisations. Worth checking whenever you are given an RSA modulus that looks suspiciously small.
Stop Skipping It
The reason crypto is the most-skipped CTF category is not that it is the hardest. Binary exploitation is harder. The reason is that it feels mathematical and unfamiliar, and that feeling triggers an avoidance response before you have even read the challenge properly.
The practical reality is that most CTF crypto challenges at beginner and intermediate level are solvable with pattern recognition, a browser, and three or four tools. The mathematics involved in RSA challenges is handled by tools that abstract it completely. What you are actually practising is the habit of reading a challenge carefully, identifying the scheme, and selecting the appropriate approach.
That habit, applied consistently, turns crypto from a category you skip into a reliable source of points.
Build the Habit on TryHackMe
The best way to get comfortable with crypto challenges before a real competition is to practise them in a low-stakes environment where you can take your time, read writeups when you are stuck, and build pattern recognition across a range of challenge types.
TryHackMe has dedicated cryptography content including the Encryption: Crypto 101 room, which covers the fundamentals of symmetric and asymmetric encryption, RSA, and hashing in a hands-on format. The broader platform gives you access to CTF-style rooms across every category so you can build crypto fluency alongside your other skills without having to search for external practice material.
Nick O'Grady