Owen Young's Blog - Zola

Adding search to zola blog with meilisearch deployed on Mogenius

2022-07-17T08:50:12+08:00

I deployed Meilisearch</a> on my VPS a couple of days ago</a>, but I’m panic, because I mess up my VPS a lot, and I rebuild my VPS a lot, so it’s not a stable environment to deploy meilisearch. So I started to browse free-for-dev</a>, see if there are some good(and cheap) solutions for me. I saw this:

mogenius</a> - A code-to-cloud platform to easily deploy any service, from static website to advanced microservice architectures. mogenius comes with fully managed hyper-scaling cloud resources, Kubernetes, CI/CD and security from Cloudflare. Free tier includes 1 CPU core, 2 GB RAM, 10 GB traffic, 4 GB SSD persistent storage. </blockquote>
The main point:

Free tier includes 1 CPU core, 2 GB RAM, 10 GB traffic, 4 GB SSD persistent storage.</li>
from static website to advanced microservice architectures.</li> </ul>
I think it’s enough for my blog’s search index. 
Mogenius</a> said they “from static website to advanced microservice architectures”, and they also have a decent plan for free tier. I decided to try it out. After completing the registration, it requires a mandatory mobile number verification. It’s a bit annoying, but I’m glad to have a free tier, and it’s a good way to avoid abuse. (There is a bug when you verify your mobile number, I’am using firefox, it seems the country code select can not show list correctly, so I just change a vpn to fix it. (maybe I should use chrome to test it, I almost give up because of this bug))
For those who want to see the final results, here is the final result:

Let’s see how to set it up:
Step 1: Create cloudspace 🔗</a></h2>
Just click ADD button, and fill in the form.
Step 2: Create a new servicee 🔗</a></h2>
Enter to the cloudspace, and click the add button to create a new service with Use a template</code>, and search meilisearch</code>
Step 3: Configure the service🔗</a></h2> Fill the servicename, for me I just type meilisearch</code> </li> Connect your github account(you can give it only selected repo permissions, you must selected at least one repo, it can be any public repo) Screenshot: </li> Click Add repository</code>, input the repo name, I input it as meili</code>, and with public</code> permission. Screenshot: </li> set resources limit (if needed, the default value is a recomendation from Mogenius). </li> set environment variables: add MEILI_MASTER_KEY</code> as key vault to protect your master key. (Click create new secret</code> to fill the secret) </li> the following environment variables can be plain text type. MEILI_NO_ANALYTICS=true MEILI_ENV=production </code></pre> All environment options are here</a> </li> </ol> Screenshot: </li> optional, set custom domain. I set it to meili.owenyoung.com</code>, I set a CNAME record meili.owenyoung.com</code> to meilisearch-prod-meilisearch-r4efxz.mo2.mogenius.io</code>, you will get the endpoint after the service created. Screenshot: </li> Then, click the Create Service</code> button. It may take 1 minute to build and deploy the service. </li> </ol> Step 5: Get your meilisearch admin key, and user key🔗</a></h2> After the service is created, the meilisearch is ready to use, you can visit https://meili.owenyoung.com/</a> to test if everything is worked fine. At this moment, the default meilisearch template version is v0.27.0rc1</code></a>, the following is the success output: { "status": "Meilisearch is running" } </code></pre> You need to run the bash script to get the admin key and user key: curl \ -X GET 'https://meili.owenyoung.com/keys' \ -H "Authorization: Bearer you-master-key" \ | json_pp </code></pre> We’ll use them in the next step. Step 6: Setup doc-scraper🔗</a></h2> We need to build search index after every updates of our blogs, I use Github Actions to build my blog, so I just add a job after the site is deployed. We’ll use docs-scraper</a> to scrape our whole site with sitemap.xml</code> file. docs-scraper</a> will help us to build heading and content level search index, it’ll bring the best search experience for our readers. The current docs-scrape is only adapted with 0.27.x</code></a>, at this moment, the latest meilisearch version 0.28 is not supported, I have submitted an issue</a>, so the default Mogenius meilisearch template version 0.27.0rc1</code> is fine, we can also upgrade it to v0.27.2</code>.(I have done this by change the Dockerfile</a>, Mongenius will monitor the updates, and automatically deploy the updates) </blockquote> See the .github/workflows/build.yml</code></a> name: Deploy Site on: workflow_dispatch: push: branches: - main jobs: build-deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - run: make install - name: Build Site run: make build - name: Deploy uses: peaceiris/actions-gh-pages@v3 with: publish_dir: ./public github_token: ${{ secrets.GITHUB_TOKEN }} scrape-and-build-search-index: needs: build-deploy runs-on: ubuntu-latest steps: # wait github pages publish - run: sleep 1m - uses: actions/checkout@v3 - name: Run docs-scraper env: HOST_URL: "https://meili.owenyoung.com" API_KEY: ${{ secrets.MEILISEARCH_API_KEY }} CONFIG_FILE_PATH: ${{ github.workspace }}/meilisearch-docs-scraper-config.json run: | docker run -t --rm \ -e MEILISEARCH_HOST_URL=$HOST_URL \ -e MEILISEARCH_API_KEY=$API_KEY \ -v $CONFIG_FILE_PATH:/docs-scraper/config.json \ getmeili/docs-scraper:v0.12.2 pipenv run ./docs_scraper config.json </code></pre> MEILISEARCH_API_KEY</code> will be the admin key from the previous step. meilisearch-docs-scraper-config.json</code></a> will be placed in the root of the repository, and it will be used by docs-scraper</code> to scrape the site. Here is my config file: { "index_uid": "owen-blog", "sitemap_urls": ["https://www.owenyoung.com/sitemap.xml"], "start_urls": [""], "scrap_start_urls": false, "stop_urls": [ "https://www.owenyoung.com/categories/", "https://www.owenyoung.com/tags/", "https://www.owenyoung.com/en/tags/", "https://www.owenyoung.com/en/categories/", "https://www.owenyoung.com/pages/", "https://www.owenyoung.com/en/pages/" ], "selectors": { "lvl0": { "selector": ".detail-page .p-category", "default_value": "Random" }, "lvl1": { "selector": ".detail-page .p-tags", "default_value": "Notes" }, "lvl2": ".detail-page .entry-title", "lvl3": ".e-content h2", "lvl4": ".e-content h3", "text": ".e-content p, .e-content li" }, "strip_chars": " .,;:#", "min_indexed_level": 2, "custom_settings": { "synonyms": { "js": ["javascript", "nodejs"] }, "stopWords": ["一", "的", "是"] } } </code></pre> Cause I’m using Chinese and English, so I added a lot of stop words of Chinese.</a> </blockquote> Then the Github Actions workflow will run docs-scraper</code> to scrape the site after the site is deployed. Step 7: Setup the frontend🔗</a></h2> See my search template</a>, basically, you need to include the docs-searchbar</a> css and js, and add an <input type="search" id="search-bar-input" placeholder="Search" /></code> to whereever you want to include the search bar. And add the following script to init: docsSearchBar({ hostUrl: "https://meili.owenyoung.com", apiKey: "your-user-api-key", indexUid: "owen-blog", inputSelector: "#search-bar-input", }); </code></pre> Conclusion🔗</a></h2> You can see the all source code on my blog source code</a>. As a programmer, it’s not a big deal to setup those things with mogenius</a>, sometimes their UI prompts are not very friendly, for example: Firstly, I tried to create meilisearch without the template, the git address I entered at the beginning was https://github.com/meilisearch/meilisearch</code>, then I can’t create the service, the error message is: This field is required.</code>, finally I tried another git url with suffix .git</code>, then the form is valid, and I can choose the git branch. I don’t know why they don’t allow me to choose a tag branch, only the normal branches are allowed, luckily, Meilisearch has one normal release branch</a>, so I can choose it.</li> Their log UI is broken on Firefox browser, the edge browser is okay.</li> Even in Edge browser, the CI/CD monitor UI is broken, with an always refreshed page.</li> </ol> Update: 2. it’s not their fault, it’s my default broswer settings that disabled the canvas rendering. </blockquote> I don’t know if they will offer a config file to setup those form, like vercel.json</code> or netlify.toml</code>, if so, it’ll be easy for other non-programmers to setup those things. Finally, I’m appreciate that they can offer a so decent free plan, I hope I don’t need to migrate it to another service. Let’s see.

Now, I'm in IndieWeb? 2022-06-12T03:40:31+08:00 Since I saw the concept of IndieWeb</a> last year, I’ve been wanting to support it on my Zola blog</a>. Before that my blog had never had a commenting system, and this time with Webmention I was finally able to get a static blog to display responses to articles on the web in plain html. I will document how I did it in this article. The IndieWeb</a> IndieWeb is a community of independent & personal websites connected by simple standards, based on the principles</a> of: owning your domain</a> & using it as your primary identity</a>, publishing on your own site (optionally syndicating elsewhere)</a>, and owning your data</a>. </blockquote> 1. Support IndieWeb Auth🔗</a></h2> IndieAuth is a federated login protocol for Web sign-in, so by enabling this, we can use our own domain to sign in to other sites and services which support IndieAuth</a>. Learn more about why IndieAuth</a> and how to IndieAuth</a>, here’s how I enabled it: By editing my index template file</a>, add this to the head:</li> </ol> <link rel="me" href="https://twitter.com/TheOwenYoung" /> <link rel="me" href="https://github.com/theowenyoung" /> <link rel="me" href="mailto:owen@owenyoung.com" /> </code></pre> Editing Twitter/GitHub profile bio, add my website URL https://www.owenyoung.com</code> </li> Test it at https://indielogin.com/</a> </li> </ol> 2. Add Profile Info🔗</a></h2> When you sign in with IndieAuth or using web mention</a>, some sites will try to get your profile info, like name, image, and bio. So we can add this to our homepage, you can learn more about h-card</a> here. Here’s how I added it: By editing my index template file</a>, add this in the aside</code> (Cause I don’t want this to affect the layout of my site, so use css display: none</code> to hide it for human, but the bot will see it): <div class="display-none h-card pt"> <img class="u-photo icon" alt="Owen" src="{{get_url(path="site/images/favicon-96x96.png",cachebust=true)}}" /> <a class="p-name u-url" href="{{ config.base_url }}" >{{ config.extra.author }}</a > <p class="p-note">{{ config.extra.bio }}</p> </div> </code></pre> 3. Joining IndieWeb Webring</a>🔗</a></h2> A webring (or web ring) is a collection of websites linked together in a circular structure, and usually organized around a specific theme, often educational or social. They were popular in the 1990s and early 2000s, particularly among amateur websites. </blockquote> Now, IndieWeb has one IndieWeb Webring</a>! by adding webring in our sites, so people can find (and be found by) other folks with IndieWeb building blocks on their sites! So, I’m in it! By joining this, my blog has been listed in the IndieWeb Webring Directory</a>, you can see I already have a profile</a> there. It’s easy to join the webring, just click the link</a>, and login with my domain www.owenyoung.com</code>, then I can get my webring code, my webring code like this: <a href="https://xn--sr8hvo.ws/%F0%9F%93%AE%F0%9F%86%99%F0%9F%93%A9/previous" >←</a > An IndieWeb Webring 🕸💍 <a href="https://xn--sr8hvo.ws/%F0%9F%93%AE%F0%9F%86%99%F0%9F%93%A9/next">→</a> </code></pre> Then, I add this code to my homepage aside</a>, you can see it on the aside footer</a> 4. Adding Webmention</a> Response to articles🔗</a></h2> Webmention</a> is a web standard for mentions and conversations across the web, a powerful building block that is used for a growing federated network of comments, likes, reposts, and other rich interactions across the decentralized social web. “An @ mention that works across websites; so that you don’t feel immovable from Twitter or Fb.” — Rony Ngala</a> </blockquote> Learn more about Webmention</a> and How to support webmention</a> Basically, I use Webmention.io</a> to collect all webmentions about this blog, and then I use Denoflow</a> to cache them to my blog repo</a>, and then I use Zola</a> load_data</code> function to load them, and render them. First, go webmention.io</a> and create a new account with my domain www.owenyoung.com</code>, then I can get my webmention endpoint, and I connected my twitter and github account to the service. </li> Second, let other services know your webmention endpoint. Add this to the head: </li> </ol> <link rel="webmention" href="https://webmention.io/www.owenyoung.com/webmention" /> </code></pre> Use Denoflow</a> to cache all webmentions to my blog repo</a></li> </ol> Workflow file(workflows/fetch-webmention.yml</code></a>): Fetch webmention API</a> to get updates, then save to webmentions</code> directory. </blockquote> sources: - use: fetch args: - https://webmention.io/api/mentions.jf2?domain=www.owenyoung.com&per-page=999&token=${{ctx.env.WEBMENTION_TOKEN}} run: return ctx.result.json() itemsPath: children key: "wm-id" filter: run: | const {ensureDir} = await import("https://deno.land/std@0.121.0/fs/mod.ts"); const { dirname } = await import("https://deno.land/std@0.121.0/path/mod.ts"); for(const item of ctx.items){ const id = item["wm-id"]; const target = new URL(item["wm-target"]); const pathname = target.pathname; const filename = pathname.slice(1).replace(/\/$/, ""); const filepath = "webmentions/"+filename+".json"; await ensureDir(dirname(filepath)); let webmentionData = {}; try { const dataString = await Deno.readTextFile(filepath); webmentionData = JSON.parse(dataString); } catch (_e) { // ignore } webmentionData[id] = item; console.log("write file:", filepath); await Deno.writeTextFile(filepath, JSON.stringify(webmentionData,null,2)); } return ctx.items.map(()=>true); </code></pre> Github Workflow file(.github/workflows/denoflow.yml</code></a>): Run denoflow every day at midnight, if there are any new updates, it will create a new pull request. </blockquote> name: Denoflow on: repository_dispatch: workflow_dispatch: # push: # branches: # - main schedule: - cron: "1 0 * * *" jobs: denoflow: runs-on: ubuntu-latest concurrency: denoflow steps: - name: Check out repository code uses: actions/checkout@v2 - uses: denoland/setup-deno@v1 with: deno-version: v1.x - run: make webmention env: WEBMENTION_TOKEN: ${{secrets.WEBMENTION_TOKEN}} continue-on-error: true - name: chown run: sudo chown -R $USER:$USER ./ - name: git config run: git config --global user.name "github-actions[bot]" && git config --global user.email github-actions-bot@users.noreply.github.com - name: git add run: git add data && git add webmentions - run: git status - id: isChanged run: git diff-index --cached --quiet HEAD || echo '::set-output name=changed::true' - name: Create pull request uses: peter-evans/create-pull-request@v3 if: ${{ steps.isChanged.outputs.changed == 'true' }} with: token: ${{ secrets.PERSONAL_TOKEN }} labels: automerge add-paths: data,webmentions commit-message: "chore: new item" committer: "github-actions[bot] <github-actions-bot@users.noreply.github.com>" author: "github-actions[bot] <github-actions-bot@users.noreply.github.com>" branch: new-item delete-branch: true title: New item update </code></pre> Github auto merge workflow file(.github/workflows/auto-merge.yml</code></a>): When a pull request is created, the workflow will automatically merge it if the pull request is from the same author </blockquote> name: Auto merge on: workflow_dispatch: pull_request_target: jobs: auto-approve: runs-on: ubuntu-latest steps: - name: Merge if: (github.actor=='theowenyoung') && (startsWith(github.head_ref,'new-item')) uses: "pascalgn/automerge-action@v0.14.3" env: GITHUB_TOKEN: "${{ secrets.PERSONAL_TOKEN }}" MERGE_DELETE_BRANCH: true MERGE_LABELS: "" </code></pre> Add response block to page tempalte</li> </ol> This part may take a while to finish, here is the main page template code</a>, I’ll only show the main part. Use Zola load_data function</a> to load the webmention data JSON file. {% set current_webmention_file_name = current_path | trim_end_matches(pat="/") %} {% set webmention_data = load_data(path="webmentions"~current_webmention_file_name~".json",required=false) %} </code></pre> Show response: {% if webmention_data %} {% set_global mentions = [] %} {% for ignored, item in webmention_data %} {% set_global mentions = mentions | concat(with=item) %} {% endfor %} <p class="muted text-sm"> {{trans(key="label_response_description",lang=lang)| markdown(inline=true) | safe}} </p> {% for type, items in mentions | group_by(attribute="wm-property") %} <h3> {{trans(key="label_"~type,lang=lang)}} ({{items | length}})<a href="#{{type}}" id="{{type}}" class="zola-anchor" >🔗</a > </h3> {% if type == 'like-of' or type == "repost-of" or type == "bookmark-of" or type == "follow-of" %} <ul class="list-none flex items-center flex-wrap"> {% for item in items %} {{ macro::webmention(type=type, item=item) }} {% endfor %} </ul> {% else %} <ul class="list-none"> {% for item in items %} {{ macro::webmention(type=type, item=item) }} {% endfor %} </ul> {% endif %} {% endfor %} {% endif %} </code></pre> Then, I add a webmention form, so if people want to submit their mentions, they can directly submit it. <form class="webmention-form" action="https://webmention.io/www.owenyoung.com/webmention" method="post" > <div class="flex items-center flex-wrap pb"> <input type="url" name="source" class="flex-3 mr-sm w-full py-sm px-sm" /> <input type="submit" class="px py-sm" value="Send Webmention" /> </div> <input type="hidden" name="target" value="{{current_url}}" /> </form> </code></pre> Optional: I also added an activity page for aggregating all response, the activity template page is here</a> </blockquote> Cause I don’t have too many mentions, so I use sebastiandedeyne’s</a> mention data as this article’s webmention data. </blockquote> 5. Send webmention when you publish a new article🔗</a></h2> When we publish a new article, we want to send a webmention to the mentioned links. We can do this by using Denoflow</a> and the Webmention.app</a> API. Webmention.app</a> can check all the mentioned links in the new article and send all webmentions to them. Before Webmention.app</a> can recognize the mentioned links, we need to add some extra microformats2</a> to our article html. Basically, it’s some html tag class names. These took me quite a few time to update my templates, so now I have supported h-card</a>, h-entry</a> and h-feed</a>. It looks like this: <article class="h-entry"> <h1 class="p-name">Microformats are amazing</h1> <p> Published by <a class="p-author h-card" href="http://example.com">W. Developer</a> on <time class="dt-published" datetime="2013-06-13 12:00:00" >13<sup>th</sup> June 2013</time > </p> <p class="p-summary">In which I extoll the virtues of using microformats.</p> <div class="e-content"> <p>Blah blah blah</p> </div> </article> </code></pre> I have updated page.html</a>, index.html</a>, taxonomy_single.html</a> and section.html</a> to support these new microformats. I have to say this is the most demanding job, good luck! Once finished, I went to indiewebify</a> to test if it can recognize my new microformats. It works! Next, I went to webmention.app</a> to apply for a token. Then I can add a denoflow workflow file to fetch it’s service every day. Here is the workflow file(workflows/send-webmention.yml</code></a>): sources: - from: https://deno.land/x/denoflow@0.0.35/sources/rss.ts args: - https://www.owenyoung.com/blog/atom.xml - from: https://deno.land/x/denoflow@0.0.35/sources/rss.ts args: - https://www.owenyoung.com/en/blog/atom.xml steps: - use: fetch args: - https://webmention.app/check?token=${{ctx.env.WEBMENTION_APP_TOKEN}}&url=${{encodeURIComponent(ctx.item.links[0].href)}} - method: GET headers: Content-Type: application/json run: | console.log(ctx.item.links[0].href); const json = await ctx.result.json(); console.log(json); </code></pre> Don’t forget to add enviroment variables WEBMENTION_APP_TOKEN</code> to your denoflow workflow file(.github/workflows/denoflow.yml</code>). - run: deno run -A https://deno.land/x/denoflow/cli.ts run env: WEBMENTION_TOKEN: ${{secrets.WEBMENTION_TOKEN}} WEBMENTION_APP_TOKEN: ${{secrets.WEBMENTION_APP_TOKEN}} </code></pre> Conclusion🔗</a></h2> I really like the concept of IndieWeb and their API design philosophy, it took some time but I still think it was worth it. It gave me renewed confidence in the Internet. You can find the all source code of this blog on Github</a> Resources🔗</a></h2> Bird.gy</a> - Bridgy connects your web site to social media. Likes, retweets, mentions, cross-posting</li> Indie Webring</a></li> Telegraph</a> - Easily send Webmentions from your website</li> Webmention.app</a> - Automate your outgoing webmentions</li> Webmention.io</a> - Webmention.io is a hosted service created to easily receive webmentions on any web page.</li> Fediring</a></li> Webmentions on a static site with GitHub Actions</a></li> </ul>