New Buzzing 已发布！

技术细节🔗</a></h2>
之前的 Buzzing 部署在 Github Pages 里，New Buzzing 迁移到了 Cloudflare 的
Pages 服务</a>：</p>
</p>
（其实为啥要搞 New Buzzing，就是因为 Github 的客服说我的 Buzzing 占的空间太大了，其中最大的一个站点占用了 60 多
G！因为之前设计的不合理，太依赖静态网站，每个 Tag，每月的 Archive 页面，甚至每个文章的详情页面都生成了
HTML，一开始还好，后来整个网站就变得越来越大，由于网站太大，用 force push 到 Github 很容易失败，所以只能正常 push，导致 repo
越积越大，直到 Github 都受不了了联系我。。而我对维护工作也变得越来越害怕）</p>
所以新版的 Buzzing，痛定思痛，决定网页全面精简化，那就是每个子网站只有 1 个页面! index.html</code>(有点吹牛，但是核心就是只有 1
个页面)，以下：</p>
.
</span>├──</span> 404.html
</span>├──</span> _redirects
</span>├──</span> en
</span>│</span>   ├── feed.json
</span>│</span>   ├── feed.xml
</span>│</span>   ├── index.html
</span>│</span>   └── lite
</span>│</span>       ├── feed.json
</span>│</span>       ├── feed.xml
</span>│</span>       └── index.html
</span>├──</span> favicon.ico
</span>├──</span> feed.json
</span>├──</span> feed.xml
</span>├──</span> icon.png
</span>├──</span> index.html
</span>├──</span> ja
</span>│</span>   ├── feed.json
</span>│</span>   ├── feed.xml
</span>│</span>   ├── index.html
</span>│</span>   └── lite
</span>│</span>       ├── feed.json
</span>│</span>       ├── feed.xml
</span>│</span>       └── index.html
</span>├──</span> lite
</span>│</span>   ├── feed.json
</span>│</span>   ├── feed.xml
</span>│</span>   └── index.html
</span>├──</span> robots.txt
</span>└──</span> zh-Hant
</span>    </span>├──</span> feed.json
</span>    </span>├──</span> feed.xml
</span>    </span>├──</span> index.html
</span>    </span>└──</span> lite
</span>        </span>├──</span> feed.json
</span>        </span>├──</span> feed.xml
</span>        </span>└──</span> index.html
</span></code></pre>
这样 Build 速度都不到 1s，目前 20 多个站点的 build 时间 30s 以内搞定（用 Deno），同时用一个动态站点i.buzzing.cc</code>
来提供不常被访问的 Tag 页面</a>，Archive
页面</a>,
以及详情页面</a>，这些页面都属于低频访问，如果每次都生成静态页面的话，会用掉大量的时间。</p>
同时，新增了www.buzzing.cc</a>作为所有子站点的聚合页面，在每次子站点生成之后生成。</p>
具体的生成过程如下，以 HN 站点为例：</p>
cache
</span>├──</span> 1-raw
</span>│</span>   └── 2022
</span>│</span>       └── 09
</span>│</span>           └── 03
</span>│</span>               └── hn
</span>│</span>                   ├── en_hn_2022_09_03_21_13_18_954_44__32700019.json
</span>│</span>                   └── en_hn_2022_09_03_21_13_18_954_45__32700066.json
</span>├──</span> 2-formated
</span>│</span>   └── 2022
</span>│</span>       └── 09
</span>│</span>           └── 03
</span>│</span>               └── hn
</span>│</span>                   ├── en_hn_2022_09_03__32696241.json
</span>│</span>                   └── en_hn_2022_09_03__32700066.json
</span>└──</span> 3-translated
</span>    </span>└──</span> 2022
</span>        </span>└──</span> 09
</span>            </span>└──</span> 03
</span>                </span>└──</span> hn
</span>                    </span>├──</span> en_hn_2022_09_03__32696241.json
</span>                    </span>└──</span> en_hn_2022_09_03__32700066.json
</span>current
</span>└──</span> items
</span>    </span>└──</span> hn
</span>        </span>└──</span> items.json
</span>archive
</span>└──</span> hn
</span>    </span>├──</span> archive
</span>    </span>│</span>   └── 2022
</span>    </span>│</span>       └── 35
</span>    </span>│</span>           └── items.json
</span>    </span>├──</span> issues
</span>    </span>│</span>   └── 2022
</span>    </span>│</span>       └── 32
</span>    </span>│</span>           └── items.json
</span>    </span>└──</span> tags
</span>        </span>└──</span> show-hn
</span>            </span>└──</span> items.json
</span></code></pre>
1-raw</code> 用来存放从Hacker News</code>或其他站点抓取的原始数据</p>
使用以下 adapter，把原始数据格式化到2-formated</code>目录下:</p>
adapters
</span>├──</span> devto.ts
</span>├──</span> googlenews.ts
</span>├──</span> hn.ts
</span>├──</span> hn_test.ts
</span>├──</span> lobste.ts
</span>├──</span> lobste_test.ts
</span>├──</span> mod.ts
</span>├──</span> newyorker.ts
</span>├──</span> ph.ts
</span>├──</span> reddit.ts
</span>├──</span> rss.ts
</span>├──</span> source.ts
</span>├──</span> twitter.ts
</span>└──</span> twitter_test.ts
</span></code></pre>
格式化后的数据长这样：</p>
{
</span>  "</span>id</span>": "</span>en_hn_2022_09_02__32677727</span>",
</span>  "</span>url</span>": "</span>https://www.commonsense.news/p/will-i-ever-see-the-36-million-oberlin</span>",
</span>  "</span>date_published</span>": "</span>2022-09-02T01:35:39.646Z</span>",
</span>  "</span>date_modified</span>": "</span>2022-09-02T01:35:39.646Z</span>",
</span>  "</span>_original_published</span>": "</span>2022-09-01T15:25:05.000Z</span>",
</span>  "</span>_original_language</span>": "</span>en</span>",
</span>  "</span>_translations</span>": {
</span>    "</span>en</span>": {
</span>      "</span>title</span>": "</span>Will I Ever See the $36MM Oberlin College Owes Me?</span>"
</span>    }
</span>  },
</span>  "</span>authors</span>": [
</span>    {
</span>      "</span>name</span>": "</span>fortran77</span>",
</span>      "</span>url</span>": "</span>https://news.ycombinator.com/user?id=fortran77</span>"
</span>    }
</span>  ],
</span>  "</span>_score</span>": </span>200</span>,
</span>  "</span>_num_comments</span>": </span>194</span>,
</span>  "</span>external_url</span>": "</span>https://news.ycombinator.com/item?id=32677727</span>",
</span>  "</span>image</span>": "</span>https://substackcdn.com/image/fetch/w_1200,h_600,c_limit,f_jpg,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Feb00ec25-b252-44f2-a06a-87cea18ea630_1440x1440.jpeg</span>"
</span>}
</span></code></pre>
我让它的格式尽量和jsonfeed</a>保持一致，同时扩展少量的自有属性。格式化后，会删除1-raw</code>的原始文件。</p>
全部格式化完成之后，进入翻译环节，翻译程序只需要读取标准的 formated 文件，然后把_translations</code>下的字段翻译成多个 target
语言，翻译后文件存放在3-translated</code>下，变成如下模样：</p>
{
</span>  "</span>id</span>": "</span>en_reddit_2022_09_02__r--Bogleheads--comments--x3bi2g--what_to_invest_in_mega_backdoor</span>",
</span>  "</span>url</span>": "</span>https://www.reddit.com/r/Bogleheads/comments/x3bi2g/what_to_invest_in_mega_backdoor/</span>",
</span>  "</span>date_published</span>": "</span>2022-09-02T01:35:40.828Z</span>",
</span>  "</span>date_modified</span>": "</span>2022-09-02T01:35:40.828Z</span>",
</span>  "</span>_original_published</span>": "</span>2022-09-01T16:07:45.000Z</span>",
</span>  "</span>_original_language</span>": "</span>en</span>",
</span>  "</span>_translations</span>": {
</span>    "</span>en</span>": {
</span>      "</span>title</span>": "</span>What to invest in mega backdoor</span>"
</span>    },
</span>    "</span>zh-Hans</span>": {
</span>      "</span>title</span>": "</span>投资于巨型后门的内容</span>"
</span>    },
</span>    "</span>zh-Hant</span>": {
</span>      "</span>title</span>": "</span>投資於巨型後門的內容</span>"
</span>    },
</span>    "</span>ja</span>": {
</span>      "</span>title</span>": "</span>メガ・バックドアに何を投資するか</span>"
</span>    }
</span>  },
</span>  "</span>tags</span>": ["</span>Bogleheads</span>"],
</span>  "</span>authors</span>": [
</span>    {
</span>      "</span>name</span>": "</span>oswestrywalesmate</span>",
</span>      "</span>url</span>": "</span>https://old.reddit.com/user/oswestrywalesmate</span>"
</span>    }
</span>  ],
</span>  "</span>_score</span>": </span>3</span>,
</span>  "</span>_num_comments</span>": </span>6</span>,
</span>  "</span>external_url</span>": "</span>https://old.reddit.com/r/Bogleheads/comments/x3bi2g/what_to_invest_in_mega_backdoor/</span>"
</span>}
</span></code></pre>
接下来进入存档环节，存档程序读取3-translated</code>下的文件，根据文件的内容，把 items
分别存放到：current</code>和archive</code>目录下，同时删除3-translated</code>下的文件：</p>
比如current/hn/items.json</code>下的文件，大概结构是这样：</p>
{
</span>  "</span>items</span>": {
</span>    "</span>en_hn_2022_09_02__32678664</span>": {
</span>      "</span>...</span>": "</span>...</span>"
</span>    },
</span>    "</span>en_hn_2022_09_02__32678665</span>": {
</span>      "</span>...</span>": "</span>...</span>"
</span>    }
</span>  }
</span>}
</span></code></pre>
这个文件会被永久存储，这就是当前网站的显示的所有条目，只存放最近 200 条数据。</p>
同时每条数据都会在这个时候归档，归档，目前是按周存放的。比如archive/hn/2022/35/items.json</code>存放第 35
周的数据，同时archive/hn/tags/show-hn/items.json</code>会存放Show HN</code>标签下最近 200
条的内容。数据格式和current/hn/items.json</code>一致。</p>
当 build hn 站点的静态网页时，只需要读取current/hn/items.json</code>下的数据，然后按照模板渲染出对应的index.html</code>,
feed.json</code>,feed.xml</code>页面，然后上传至 cloudflare pages 即可。</p>
对于动态页面，比如https://i.buzzing.cc/hn/tags/show-hn/</code>,
会实时读取archive/hn/tags/show-hn/items.json</code>下的数据，然后按照模板渲染。</p>
目前动态页面的程序部署在Deno Deploy</a>上，current</code>,archive</code>,cache</code>文件存放在Cloudflare R2</a>上，静态网页部署在Cloudflare Pages</a>上。同时使用
Github Actions 来运行定时任务，每半个小时运行一次，20 多个站点，目前每次仅需要 3
分多钟就能完成全部的更新，翻译，发布静态网页。对比以前，抓取数据的时间差不多，但是构建和发布一个静态网站的时间就要多达 30 分钟。</p>
具体的 Github Actions 配置如下：</p>
steps</span>:
</span>  - </span>name</span>: </span>Check out repository code
</span>    </span>uses</span>: </span>actions/checkout@v3
</span>  - </span>uses</span>: </span>actions/setup-node@v3
</span>    </span>with</span>:
</span>      </span>node-version</span>: </span>16
</span>      </span>cache</span>: "</span>npm</span>"
</span>  - </span>name</span>: </span>install wrangler
</span>    </span>run</span>: </span>npm install -g wrangler
</span>  - </span>uses</span>: </span>denoland/setup-deno@v1
</span>    </span>with</span>:
</span>      </span>deno-version</span>: </span>v1.x
</span>  - </span>uses</span>: </span>actions/cache@v3
</span>    </span>with</span>:
</span>      </span>path</span>: </span>|
</span>        ~/.deno
</span>        ~/.cache/deno
</span>      </span>key</span>: </span>${{ runner.os }}-deno-${{ hashFiles('**/*deps.ts') }}
</span>  - </span>run</span>: </span>make prod-load
</span>    </span>env</span>:
</span>      </span>AWS_ACCESS_KEY_ID</span>: </span>${{ secrets.AWS_ACCESS_KEY_ID }}
</span>      </span>AWS_SECRET_ACCESS_KEY</span>: </span>${{ secrets.AWS_SECRET_ACCESS_KEY }}
</span>      </span>AWS_DEFAULT_REGION</span>: </span>${{secrets.AWS_DEFAULT_REGION}}
</span>      </span>AWS_ENDPOINT</span>: </span>${{secrets.AWS_ENDPOINT}}
</span>  - </span>run</span>: </span>make install
</span>  - </span>run</span>: </span>make prod-source
</span>    </span>id</span>: </span>source
</span>    </span>continue-on-error</span>: </span>true
</span>    </span>env</span>:
</span>      </span>AWS_ACCESS_KEY_ID</span>: </span>${{ secrets.AWS_ACCESS_KEY_ID }}
</span>      </span>AWS_SECRET_ACCESS_KEY</span>: </span>${{ secrets.AWS_SECRET_ACCESS_KEY }}
</span>      </span>AWS_DEFAULT_REGION</span>: </span>${{secrets.AWS_DEFAULT_REGION}}
</span>      </span>AWS_ENDPOINT</span>: </span>${{secrets.AWS_ENDPOINT}}
</span>      </span>TWITTER_BEARER_TOKEN</span>: </span>${{secrets.TWITTER_BEARER_TOKEN}}
</span>      </span>TWITTER_ACCESS_TOKEN</span>: </span>${{secrets.TWITTER_ACCESS_TOKEN}}
</span>      </span>TWITTER_ACCESS_TOKEN_SECRET</span>: </span>${{secrets.TWITTER_ACCESS_TOKEN_SECRET}}
</span>      </span>TWITTER_CONSUMER_SECRET</span>: </span>${{secrets.TWITTER_CONSUMER_SECRET}}
</span>      </span>PRODUCTHUNT_TOKEN</span>: </span>${{secrets.PRODUCTHUNT_TOKEN}}
</span>      </span>TRANSLATE_TOKEN</span>: </span>${{secrets.TRANSLATE_TOKEN}}
</span>  - </span>name</span>: </span>upload files
</span>    </span>run</span>: </span>make prod-upload
</span>    </span>env</span>:
</span>      </span>AWS_ACCESS_KEY_ID</span>: </span>${{ secrets.AWS_ACCESS_KEY_ID }}
</span>      </span>AWS_SECRET_ACCESS_KEY</span>: </span>${{ secrets.AWS_SECRET_ACCESS_KEY }}
</span>      </span>AWS_DEFAULT_REGION</span>: </span>${{secrets.AWS_DEFAULT_REGION}}
</span>      </span>AWS_ENDPOINT</span>: </span>${{secrets.AWS_ENDPOINT}}
</span>      </span>DUFS_URL</span>: </span>${{secrets.DUFS_URL}}
</span>      </span>DUFS_SECRETS</span>: </span>${{secrets.DUFS_SECRETS}}
</span>  - </span>name</span>: </span>throw if source failed
</span>    </span>if</span>: </span>steps.source.outcome == 'failure'
</span>    </span>run</span>: </span>|
</span>      echo "::error::Source build failed"
</span>      exit 1
</span>  - </span>name</span>: </span>Publish pages
</span>    </span>run</span>: </span>make prod-publishall
</span>    </span>env</span>:
</span>      </span>CLOUDFLARE_ACCOUNT_ID</span>: </span>${{secrets.CLOUDFLARE_ACCOUNT_ID}}
</span>      </span>CLOUDFLARE_API_TOKEN</span>: </span>${{secrets.CLOUDFLARE_API_TOKEN}}
</span></code></pre>
最后，使用Pipedream</a>,每一个小时拉取 feedjson，然后分别发布到
Twitter：@HackerNewsZh</a>,
@NewsBotZh</a>,
@AskRedditZh</a>.</p>
小细节🔗</a></h2>
Youtube 视频的懒加载🔗</a></h3>
如果你引入 Youtube 视频的全部 Embed 代码，那么你的网页每次会加载 Youtube
内部超多的资源，这很不道德，所以我们可以用iframe</code>的srcdoc</code>属性来实现只加载一个预览页面，而不实际加载整个
iframe。当用户主动点击播放的时候，这个时候才去真正的 loading,效果如下：</p>
</p>
具体代码如下：</p>
<</span>iframe
</span>  </span>class</span>="</span>embed-video</span>"
</span>  </span>loading</span>="</span>lazy</span>"
</span>  </span>src</span>="</span>https://www.youtube.com/embed/${embedUrlVideoId}&autoplay=1</span>"
</span>  </span>srcdoc</span>="</span><style>*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img,span{position:absolute;width:100%;top:0;bottom:0;margin:auto}span{height:1.5em;text-align:center;font:48px/1.5 sans-serif;color:white;text-shadow:0 0 0.5em black}</style><a href=https://www.youtube.com/embed/${embedUrlVideoId}?autoplay=1><img src=https://img.youtube.com/vi/${embedUrlVideoId}/hqdefault.jpg loading='lazy' alt='Youtube Preview Image'><span>▶</span></a></span>"
</span>  </span>frameborder</span>="</span>0</span>"
</span>  </span>allow</span>="</span>accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture</span>"
</span>  </span>allowfullscreen
</span>></</span>iframe</span>>
</span></code></pre>
总结🔗</a></h2>
用 Deno 做完这一切之后，只能用神清气爽来形容（可以说这是我最满意的一次应用结构设计）。再也不怕新增站点了！ Deno 开箱即用的 Typescript
支持，以及 Deno Deploy 的无缝部署都让我非常享受，同时 Deno
生态的不完整性，也让我更少的依赖别人的东西，整个网站做下来感觉非常的健壮。也许谁有时间可以利用站点提供的feed.json</code>来生成一个体验良好的 APP.</p>

Owen的博客 - Buzzing

New Buzzing 已发布！