Factoring polynomials is simply the reversal of the process of multiplying polynomials. While there is no single technique we can use to do this to an arbitrary polynomial, there are several tricks that can help.
Factorization rules for the squares and cubes of binomials can be obtained by simply expanding the related products, as shown below
Reversing these results, we arrive at the factorization rules:
$$\boxed{\displaystyle{\begin{array}{rcl} x^2 + 2xy + y^2 &=& (x+y)^2\\ x^3 + 3x^2y + 3xy^2 + y^3 &=& (x+y)^3 \end{array}}}$$When we see polynomials that follow one of the above patterns, factoring them becomes easy. For example,
$$\begin{array}{rcl} 9a^2 + 12ab^3 + 4b^6 &=& (3a)^2 + 2(3a)(2b^3) + (2b^3)^2\\\\ &=& (3a + 2b^3)^2 \end{array}$$If we work under the assumption that this can break into two or more factors, one thing we might try is to consider different integer values of $x$ and $y$. If we can find a consistent way to factor the resultant integer value of $x^2-y^2$, this might hint at a way to factor the polynomial itself. With this in mind, consider the following table for some randomly† chosen values of $x$ and $y$: $$\begin{array}{|c|c|c|c|} x & y & x^2 - y^2 & \textrm{integer factors}\\\hline 5 & 2 & 21 & 7 \cdot 3\\\hline 7 & 4 & 33 & 11 \cdot 3\\\hline 10 & 3 & 91 & 13 \cdot 7\\\hline 11 & 6 & 85 & 17 \cdot 5\\\hline 27 & 14 & 533 & 41 \cdot 13\\\hline \end{array}$$
Can you guess a formula for the integer factors in terms of the values of $x$ and $y$? (Try!)
Note how in the first line above, $7 = 5 + 2$ while $3 = 5 - 2$. Likewise, in the second line, $11 = 7 + 4$ and $3 = 7 -4$. It would appear that one can always factor $x^2 - y^2$ into the product of the sum of $x$ and $y$ and the difference of the same. That is to say, it appears that $$\boxed{\displaystyle{x^2 - y^2 = (x+y)(x-y)}}$$
Expanding the right side (using F.O.I.L to shorten the calculation) quickly verifies this guess is correct:
$$\begin{array}{rcl} (x+y)(x-y) &=& x^2 -xy + xy - y^2\\ &=& x^2 - y^2 \end{array}$$Of course, now this result serves as a template that helps us factor other polynomial expressions.
Notice for example $16a^2b^6 - 49$ is also a difference of two squares: $(4ab^3)^2$ and $7^2$. Thus $4ab^3$ plays the role of $x$ and $7$ plays the role of $y$. The analogs of $(x+y)$ and $(x-y)$ then tell us that
$$16a^2b^6 - 49 = (4ab^3 + 7)(4ab^3 - 7)$$Often in mathematics one finds it useful to explore multiple paths to a given solution. In the discussion above, we essentially made an educated guess and then verified that guess with some algebra. Notice how we could have also argued the result directly by adding a well-chosen value of zero and then factoring two groups of terms to identify a common factor:
$$\begin{array}{rcl} x^2 - y^2 &=& x^2 + \overbrace{xy - xy}^{zero} - y^2\\\\ &=& (x^2 + xy) + (-xy - y^2) \quad \scriptsize{\textrm{grouping the first two and last two terms}}\\\\ &=& x(x+y) + (-y)(x+y) \quad \scriptsize{\textrm{after factoring each group}}\\\\ &=& (x+y)(x + (-y)) \quad \scriptsize{\textrm{the common } (x+y) \textrm{ is factored out}}\\\\ &=& (x+y)(x-y) \end{array}$$This technique used above of splitting a polynomial into groups of terms, and then identifying a factor common to all of the groups can be helpful in factoring polynomials more generally, as the following three examples suggest:
$$\begin{array}{rcl} x^2 - 2xy + 5x - 10y &=& (x^2 - 2xy) + (5x - 10y)\\\\ &=& x(x-2y) + 5(x-2y)\\\\ &=& (x-2y)(x+5)\\\\\\ x^2 + 3x - 3y - y^2 &=& (x^2 - y^2) + (3x - 3y) \quad \scriptsize{\textrm{notice we had to move some terms around}}\\\\ &=& (x+y)(x-y) + 3(x-y)\\\\ &=& (x-y)(x+y+3)\\\\\\ a^2b + 2a^2c + ab + 2ac + b + 2c&=& (a^2b + ab + b) + (2a^2c + 2ac + 2c) \quad \scriptsize{\textrm{again we moved some terms around}}\\\\ &=& b(a^2 + a + 1) + 2c(a^2 + a + 1)\\\\ &=& (a^2 + a + 1)(b + 2c) \end{array}$$We know how to factor the difference of squares now -- so how about the difference of cubes?
Here again, the addition of a well-chosen value of zero helps us complete the task via factoring by grouping:
$$\begin{array}{rcl} x^3 - y^3 &=& x^3 + \overbrace{-x^2y + x^2y}^{\textrm{zero}} - y^3\\\\ &=& (x^3 - x^2y) + (x^2y - y^3)\\\\ &=& x^2(x-y) + y(x^2-y^2)\\\\ &=& x^2(x-y) + y(x+y)(x-y)\\\\ &=& (x-y)(x^2 + y(x+y))\\\\ &=& (x-y)(x^2 + xy + y^2) \end{array}$$The sum of cubes can be factored using a very similar process, giving us in total two new factorization rules:
$$\boxed{\displaystyle{\begin{array}{rcl} x^3 - y^3 &=& (x - y)(x^2 + xy + y^2)\\ x^3 + y^3 &=& (x + y)(x^2 - xy + y^2) \end{array}}}$$Just as before, the above gives us two more templates for factoring other polynomials. For example, if one was attempting to factor $a^3b^6 + 8$, one might notice that the first term is the cube of $ab^2$ and the second is the cube of $2$. Using these values in the above rules as our $x$ and $y$ respectively, we discover
$$\begin{array}{rcl} a^3b^6+8 &=& (ab^2)^3 + 2^3\\\\ &=& (ab^2 + 2)((ab^2)^2 - (ab^2)\cdot 2 + 2^2)\\\\ &=& (ab^2 + 2)(a^2b^4 - 2ab^2 + 4) \end{array}$$Factoring a trinomial that is not the square of a binomial and whose terms don't have a common monomial factor requires a bit of detective work.
Suppose we wish to factor a trinomial of the form $x^2 + bx + c$, where $b$ and $c$ are integers. Let us make a guess that $x^2 + bx + c = (x + r)(x + s)$. Expanding we see $x^2 + bx + c = x^2 + (r + s)x + r \cdot s$. As such, we seek values of $r$ and $s$ so that $r+s = b$ and $r \cdot s = c$.
Let us illustrate how we might find these values of $r$ and $s$ with an example:
Suppose we want to factor $x^2 - 5x + 6$. Thus, we need $r$ and $s$ so that $r+s = -5$ and $r \cdot s = 6$. Given that $r \cdot s$ must be positive, we notice $r$ and $s$ must have the same sign.
Now consider possible integer factorizations of $6$, namely:
$$\begin{array}{rcr} 1 &\times& 6\\ -1 &\times& -6\\ 2 &\times& 3\\ -2 &\times& -3 \end{array}$$When one adds two numbers of the same sign, one adds their magnitudes and keeps the same sign on the result. Ergo, $r$ and $s$ must be given by a product where the magnitudes of the numbers involved sum to $5$. Thus, the magnitudes involved must be $2$ and $3$.
Given that $r+s = -5$ and $r$ and $s$ have the same sign, we further conclude $r$ and $s$ must both be negative. As such, we make a guess $r = -2$ and $s = -3$. Expanding $(x-2)(x-3)$ confirms this guess is correct:
$$\begin{array}{rcl} (x-2)(x-3) &=& x^2 -3x -2x + 6\\\\ &=& x^2 - 5x + 6 \end{array}$$Now suppose we wish to factor a trinomial whose leading coefficient is not equal to one -- something of the form $ax^2 + bx + c$. Things are a bit more complicated here. One way to proceed would be to first make a guess that the factorization takes the form $(px + r)(qx + s)$. Expanding this product, we notice
$$\begin{array}{rcl} (px+r)(qx+s) &=& pqx^2 + psx + qrx + rs\\\\ &=& pqx^2 + (ps + qr)x + rs \end{array}$$Thus, $ax^2 + bx + c = pqx^2 + (ps + qr)x + rs$. Notice that the product of the leading coefficient and the constant term is given by both $ac$ and $pqrs$. The product of the two values in the parentheses is also $pqrs$. As such, we seek two values, $pq$ and $rs$ whose product is $ac$ and whose sum is $b$.
Let us demonstrate the rest of the technique with an example...
Suppose we wish to factor $2x^2 + 7x + 6$. We then seek two values $pq$ and $rs$ whose product is $12$ and whose sum is $3$. The only integer possibilities are, of course, $3$ and $4$. This gives us enough information to finish the factorization by grouping:
$$\begin{array}{rcl}
2x^2 + 7x + 6 &=& 2x^2 + (4x + 3x) + 6\\\\
&=& (2x^2 + 4x) + (3x + 6)\\\\
&=& 2x(x+2) + 3(x+2)\\\\
&=& (2x+3)(x+2)
\end{array}$$
While sometimes more creative strategies need to be applied, following (in order) the list below of actions one can take often results in a quick factorization of a given expression.
An expression is considered to be completely factored when none of its factors can themselves be non-trivially factored. For a complete factorization, the actions in the list below should be repeatedly applied to the resulting factors until they are no longer useful.
Factor out the any and all common monomial factors
Check for special products
Try factoring by other methods
†: OK, you got me -- the values in the table weren't chosen completely at random. They were chosen to make guessing the pattern a bit easier. Note, a polynomial factorization will certainly give us a way to write an integer of that form as a product, but some integers can be written as a product in more than one way. For example, $$7^2 - 3^2 = 40 = 10 \cdot 4 = 20 \cdot 2 = 5 \cdot 8 = 1 \cdot 40$$ For our purposes, all but the first factorization (i.e., $10 \cdot 4$) are essentially distractions. The numbers in the table were chosen so that such "distractions" were minimized.