On-Page SEO Basics: Meta Tags, Robots.txt, and Favicons

Good content can still be held back by a handful of technical details: a title that gets cut off, a canonical that points to the wrong URL, a robots.txt rule that doesn't do what its author thought, or a favicon search results can't use. None of these take long to get right. They just have to be right.

The tags in <head> and what each one does

This is the block the Meta Tag Generator produces for a page:

<title>GST Calculator for Invoices | Acme Tools</title>
<meta name="description" content="Add or remove GST at 5%, 18% or 40% and see the base, tax and total instantly." />
<link rel="canonical" href="https://acme.example/gst-calculator" />

<!-- Open Graph -->
<meta property="og:type" content="website" />
<meta property="og:title" content="GST Calculator for Invoices | Acme Tools" />
<meta property="og:description" content="Add or remove GST at 5%, 18% or 40% and see the base, tax and total instantly." />
<meta property="og:url" content="https://acme.example/gst-calculator" />
<meta property="og:image" content="https://acme.example/og/gst.png" />
<meta property="og:site_name" content="Acme Tools" />

<!-- Twitter -->
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:title" content="GST Calculator for Invoices | Acme Tools" />
<meta name="twitter:description" content="Add or remove GST at 5%, 18% or 40% and see the base, tax and total instantly." />
<meta name="twitter:image" content="https://acme.example/og/gst.png" />

• Title. This is the blue link in search results. Results are cut off by pixel width rather than a character count, which in practice is roughly 50–60 characters. Put the words that describe the page first and the brand last, so truncation removes the brand, not the meaning.
• Description. It doesn't affect rankings, but it's often the snippet people read before deciding to click. Search engines sometimes rewrite it using text from the page, especially when the one you wrote doesn't match the query. Keep it around 150–160 characters and specific to the page.
• Canonical. This tells search engines which URL is the real one when the same content is reachable at several addresses (with and without a trailing slash, with tracking parameters, and so on). It must be absolute, and it must point to a URL that returns 200. A canonical pointing at an address that redirects elsewhere sends contradictory signals. A September 2026 audit of this site found exactly that on almost every page: canonicals and sitemap entries used addresses without a trailing slash, and the server redirected every one of them to the slash version. Fixing the server so those exact URLs answered directly was the first change made.
• Open Graph and Twitter tags. These control link previews in messaging apps and social networks. og:image must be an absolute URL, and around 1200 × 630 pixels suits large previews. If the twitter: tags are missing, X falls back to the og: versions.

robots.txt controls crawling, not indexing

A Disallow rule asks well-behaved crawlers not to fetch a path. It does not stop the URL from being indexed. If other pages link to a blocked URL, it can still show up in search results, usually with no description, because the crawler was never allowed to read it.

So to keep a page out of search results:

1. Add <meta name="robots" content="noindex"> to the page, or send the equivalent X-Robots-Tag HTTP header.
2. Leave the page crawlable in robots.txt. If crawlers can't fetch the page, they never see the noindex.

That second step is where people get it backwards, blocking a page in robots.txt "to hide it" and so guaranteeing the noindex is never read. This site follows the correct pattern for the pages it deliberately keeps out of search. They carry noindex, follow, they're omitted from the sitemap, and robots.txt doesn't block them.

Use robots.txt for what it's good at: keeping crawlers out of endless internal-search results, faceted-filter URLs, and admin areas, so they spend their time on pages that matter.

Writing robots.txt correctly

The Robots.txt Generator builds the file from a user-agent, Disallow and Allow paths, an optional crawl delay, and a sitemap URL:

User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /search/help

Sitemap: https://acme.example/sitemap.xml

Details that matter:

• One file per host, at the root. https://acme.example/robots.txt doesn't apply to https://www.acme.example or https://shop.acme.example. Each needs its own file.
• Paths are case-sensitive, and they match by prefix: Disallow: /search also blocks /search-results.
• The most specific rule wins in Google. Allow: /search/help above re-opens one path under a blocked prefix.
• `Disallow:` with nothing after it allows everything, while Disallow: / blocks everything. They're one character apart, so double-check before you deploy.
• Google ignores `Crawl-delay`. Bing and some other crawlers honour it. Google adjusts its crawl rate automatically instead.
• The `Sitemap:` line takes a full absolute URL and can appear anywhere in the file.

Favicons that show up in search results

Browsers are forgiving about favicons. Search results are not: Google's favicon guidelines ask for a square favicon of at least 8 × 8 pixels, recommend one larger than 48 × 48, and need it linked from the homepage and crawlable. The Favicon Generator turns one image into 16 × 16, 32 × 32 and 48 × 48 PNGs, plus the 180 × 180 apple-touch-icon.png that iPhones use for home-screen bookmarks. It has controls for shape, background colour and padding, since a detailed logo usually needs simplifying to stay legible at 16 pixels. Reference them like this:

<link rel="icon" type="image/png" sizes="48x48" href="/favicon-48x48.png" />
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png" />
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png" />
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png" />

A quick check after deploying

Open your live page's source and paste it into the HTML Meta Tag Extractor. It lists what crawlers will actually see: the title, description, canonical, every Open Graph and Twitter tag, and any robots meta tag, including a forgotten noindex from a staging build. The meta-tag extraction guide explains why, on JavaScript-heavy sites, the page source and the rendered page can disagree.