Tag: bots

Posted on
by

My posting out-put has been subpar of late, but today the Pook-Emu Bee links return for the first time since September 10.

1 | 25 Years Since the Worst Day in My Lifetime (Andrew R. Arthur for Center for Immigration Studies. Sep. 11, 2026.)
Memories of 9/11 and border/visa security in the ensuing quarter-century.

2 | New World Trade Center is fine but it could have been great (Steve Cuozzo for the New York Post. Sep. 12, 2026.)
Given the number of stupid ideas that were floating around in the wake of 9/11 (many of which are noted in this op-ed), I am willing to count the “fine” Freedom Tower as a “W.”

3 | What 9/11 was like for veteran Post police reporter: ‘Nothing could prepare me’ (Larry Celona for the New York Post. Sep. 11, 2026.)
Memories from a repporter who had covered 9/11 on the ground.

4 | The Second Life of Crayons (Dan Lewis for Now I Know. May 15, 2019.)
Innovative crayon recycling.

5 | Guess Who Helped Doxx the Holdout Juror in the Lindsay Clancy Case (Victoria Taft for PJ Media. Sep. 13, 2026.)
Someone ought to look into whether attorney Reddington and his wife have been communicating with some of the “11” subsequent to the trial.

6 | 220 million traveler records exposed in Vietnam-linked APIS leak (Ax Sharma for Bleeping Computer. Sep. 8, 2026.)
I’m 100% sure I am not part of this leak thanks to one simple trick.

7 | Bernie Sanders proposes 20 year prison sentence for AI devs who plow ahead with Artificial Superintelligence plans — penalty on par with illegally developing rogue nuclear weapons (Mark Tyson for Bleeping Computer. Sep. 13, 2026.)
Superintelligence relative to what? If the bar for human intelligence is this proposal than the original Mario Party devs are in trouble.

8 | Publishing your work increases your luck (Aaron Francis for The ReadME Project. Juuly 12, 2022.)
Every Pook-Emu Bee is a roll of the dice.

9 | You Can Drop SEO (Ibrahim Diallo at iDiallo. Sep. 12, 2026.)
Not here I can’t. The “brand SEO” of “Pook-Emu Bee” is too elite.

10 | Feds accuse China of ‘systematic’ distillation of U.S. AI models (Derek B. Johnson for Cyberscoop. Sep. 8, 2026.)
To be fair to China… a decent number try to scrape my sites directly

Posted on
by

I discovered two new bots in my server logs last week: RSSAmplifierBot and BriskBot. They come from RSS Amplifier and Brisk News respectively. Bot sites are built by Profullstack, Inc. I discovered that RSS Amplifier, unbeknownst to me, had been crawling New Leaf Journal’s ATOM feed and re-publishing (albeit they would not describe it as such) the entire contents of my article under the RSS Amplifier banner (note it appears it took entire articles because I offer full-text feeds; it had truncated entries for sites that do not offer full text feeds). While the articles are attributed with links back to my original source, I do not approve of wholesale re-publishing (as should have been clear from my About Page). Worse yet, RSS Amplifier describes its purpose in part as making articles from around the web readily available for AI crawlers (live About Page) and they sell access to their data (live Training Data page). Brisk News appears to host articles from around the web with AI summaries, but there is less information about the project on site.

I present this information for the consideration of my fellow webmasters. I took it upon myself to block both RSS Amplifier and Brisk News at the server level and have a pending take-down request with RSS Amplifier. Alternatively, a webmaster who wants to be part of the collection may want to test whether the team responds to requests about data handling (RSS Amplifier does insist you can make a request to be in the directory but not have your site’s data sold, but I have my doubts on how that would work in practice). In any event, for those who are interested in learning more:

Posted on
by

Pook-Emu Bee links are so real now that we’re delivering on Sundays.

Statue of Mary casting a shadow against the stone wall of a cathedral.
I took this photograph outside the Our Lady of Lebanon Maronite Cathedral in Brooklyn Heights on August 6, 2026.

1 | Tracking Bees with a Stereo Camera (Al Williams for Hackaday. August 8, 2026.)
A fitting entry in light of today’s New Leaf Journal review of A Taste of Honey.

2 | RV Steelhead (Tugster: A Waterblog. August 8, 2026.)
Survey research of a survey vessel.

3 | Why some tech users are ditching Android for Linux phones (Maya Brooks for Northeast Times. August 6, 2026.) (HT Tux Machines.)
This reminds me that I need to write about my experiences with Ubuntu Touch… and by extension my return to LineageOS.

4 | Mamdani taps Israel-hating, anti-cop radicals for NYC school board (David Spector for the New York Post. August 8, 2026.)
The headline is accurate but it leaves out the weird overcompensating dude who wants to create race-segregated discussion groups (and pursued that agenda while working in DOE during the de Blasio administration).

5 | Most Googlebots are fake (Bobbie Chen for Digital Seams. July 26, 2026.) (HT Lazy Reading.)
I don’t doubt it but I will submit that to the best of my recollection, NLJ was never brought to heel by a bot falsely advertising itself as Googlebot. Most of the bots that hit NLJ aggressively enough to take it down had generic user agents. In one case it was Scrapy.

6 | Linux 7.2 Drops Ancient PROFIBUS Driver: Ported From SCO Unix In 1998, Unused For Years (Michael Larabel for Phoronix. June 25, 2026.)
I never heard of this driver but dropping it from the Linux kernel both disgusts and offends me. I’m switching to Windows 98.

7 | Linux gaming got so big that even GOG couldn’t ignore it anymore (Simon Batt for XDA Developers. August 7, 2026.)
I approve of the respect for Linux. With that being said, despite my having a collection of GOG games, I will probably stick with using Lutris since I use Lutris as a WINE front-end for some non-GOG games.

8 | Woke Canadian deported for slapping teens wearing MAGA clothing faces one final humiliation as she’s banished from the US (Eliot Force for Daily Mail. August 5, 2026.)
I wonder what her lawyer thought when he realized that he had so little to work with that he was going to have to go with her having engaged in unauthorized employment as being a positive equity.

9 | Have an Intel Mac or Old Windows PC? Revive It With One of These 20 Fun DIY Projects (Eric Griffin for PC Mag. August 5, 2026.)
Suggestions 1-3 and 6-7 are good. I’d throw in making a designated machine for retro games, albeit that is arguably a subset of 1 and/or 2, Starts grasping at straws toward the end, but points for creativity.

Posted on
by

I was only able to publish two editions of Pook-Emu Bee links last week. Thus, to make amends, I present a rare Sunday edition of the Pook-Emu Bee. For link fans out there, I include many links in my weekly newsletter. See my June 6, 2026 Newsletter for an example.

Photograph of a bush with green leaves and brown berries in Brooklyn Heights. Out of focus in the background are brick townhouses.
I took this photograph of a bush with berries in Brooklyn Heights on the morning of June 7, 2026.

1.Bug Fables: Almost, but Not Quite, Paper Mario 3.0 (Wouter Groeneveld at Jefklak’s Codex. November 29, 2021.)

I never heard of this Paper Mario-inspired RPG, but as a big Paper Mario fan, consider me intrigued. Add it to the ever-growing to-do list.

2. ‘Bots have now passed human traffic online,’ Cloudflare boss laments — says agentic traffic wasn’t expected to eclipse real people until next year (Mark Tyson for Tom’s Hardware. June 4, 2026.)

I am reasonably confident that I have never had more human traffic than bot traffic save for the peak of a few Hacker News page one surges.

3. I made a bee bath (Laura Michet. June 6, 2026.)

That’s a good bee bath. There is still plenty of time for it to welcome bee visitors.

4. 98-year-old beaten with broomstick, chair in NYC by woman campaigning for Dem: cops ( Sofia Poznansky, Marie Pohl, Rich Calder and Tina Moore for the New York Post. June 6, 2026.)

The campaigns aren’t hiring the best.

5. Chet Holmgren and the $240 Million Disappearing Act (Kevin O’Connor. May 31, 2026.)

I did not watch but I assess that you would generally prefer that your maximum contract guys do not disappear in win-or-go-home games.

6. Anthropic warns AI could soon help build its own successors (Ashley Gold for Axios. June 4, 2026.)

I respect Antrhopic’s ability to just post stuff that gets turned into free advertising my various news outlets.

7. Richard Gere Says He’s ‘Deeply Ashamed’ That Trump Admin Is Calling Migrants ‘Aliens’ During Speech in Germany (Alana Mastrangelo for Breitbart. June 6, 2026.)

The trouble with actors begins when they don’t have a script to read from. But fear not: I offer a script in the form of my 2022 article, Aliens vs Noncitizens – The Anatomy of a Debate. By the way, that headline I came up with back in 2022 looks vaguely AI. But rest assured, everything from the headline to the article is 100% original, human grown “content.”

Posted on
by

I checked my emails this morning and saw an alert from Cloudron that New Leaf Journal had become non-responsive. By the time I saw the report, NLJ had recovered. I was a bit alarmed since I had recently increased my server resources and implemented a new server-level IP blocking strategy. I checked the logs and found the obvious culprit, which flooded NLJ with requests and then kept making requests even after NLJ went down. Unsurprisingly, I was not the only one to complain about this IP. Get your bad actors in order, TECHOFF SRV LIMITED.

Posted on
by
Replied to Observations on blocking various webbots by Sean Conner (The Boston Diaries)
Going through the logs from my web server for March, I noticed that 26% of all requests resulted in a failed client request (stuff like “404 Not Found” or “429 Too Many Requests”). These requests are more annoying than they are debilitating, but ideally, I would love a way to crash these bots as they're mostly scanning my site for exploits; fully 50% are just scanning for various PHP based scripts (which I don't use at all) and the rest for a variety of other files that can lead to exploits.

Sean Conner at The Boston Diaries wrote about trying to block annoying and/or malicious bots from crawling his website. It is a good read.

Beginning in late 2024, I started noticing The New Leaf Journal going down periodically. The server logs suggested that it was being overrun by bots and crawlers. I tried various methods to ensure that NLJ would be up all the time, including fiddling with Apache configs and my .htaccess file. I stumbled upon the “solution” (for now, at least) in late January. I use Cloudron to manage this server (which is home to both this site and NLJ). Cloudron has a built-in firewall. Back on January 30, I published Using Cloudron’s IP Firewall. I populated its firewall with IPs from the AbuseIPD blacklist, which is readily available in LittleJake’s GitHub repository. NLJ’s resource usage went way down (see the chart in my linked post) and my crashing issues have been gone for more than two months. The AbuseIPD blacklist is updated approximately every six hours. I have been adding the IPs manually but I have been meaning to look into a way to automate the process.

Posted on
by

I just happened to have my NLJ logs open (I had opened them when the site was slow for a moment). I saw something called the YandoriRSSBot requesting the NLJ ATOM feed. While not unprecedented, almost all the feed fetchers ask for the regular RSS feed. I decided to search for the user agent to see if it is coming from a new feed reader that I had never heard of. Unfortunately, Known Agents has no information about it beyond the fact that it has been reported in the wild. But I ran another search and found a discussion about “Yandori” on Hacker News. From the creator:

I built a system that monitors ~200,000 news RSS feeds in near real-time and clusters related articles to show how stories spread across the web. It uses Snowflake’s Arctic model for embeddings and HNSW for fast similarity search. Each “story cluster” shows who published first, how fast it propagated, and how the narrative evolved as more outlets picked it up.

User antiochIst on Hacker News

Interesting. While it is not something I would use, I shall allow it so long as it behaves (three consecutive requests in a few seconds for my ATOM feed and a fourth a couple of minutes later seems a bit off, but nothing too alarming in the grand scheme of things.)

Posted on
by

I wanted to add an htaccess rule to NLJ blocking requests from empty user agents (and “-” user agengs). I looked into the best way to do it and found a 2017 answer on Stack Overflow. This solution is interesting because it explicitly allows requests to the site’s feed. While I have not noticed empty user agent requests for our feeds, there are many niche feed readers out there, so I went with a modified version of this snippet to be on the safe side.

Posted on
by

This site is hosted on the same server as The New Leaf Journal, and I use Cloudron to manage the server. Beginning around early spring last year, The New Leaf Journal began to crash once every few days when Cloudron health checks failed (I have not had any similar issues with this site, which is much lower traffic). I attributed the problems to bot traffic, and I think I was correct to do so. I tried many fixes, some of which worked for a time before the downtime problems returned. After having a few too many incidents in January, I bit the bullet and moved up a tier on our Hetzner VPS server (I had been on the higher resource tier prior from February 2022 through May 2023 before downgrading to a smaller VPS to save a small amount money, initially without any problems). I thought all was well until NLJ crashed again for three hours yesterday morning. I am not inclined to go up another tier, so I decided to see if I could find a more durable solution that did not involve changing how I run NLJ (I do not have time to sit down and work on that now) or spending more money (I would rather eat the occasional downtime at a point). I poked around and noticed that Cloudron has a built-in firewall separate from app-level settings (I use Jeff Starr’s 8G Firewall in the .htaccess files for both this site and NLJ for app-level protection). I figured that preventing the bots from hitting NLJ at all may reduce strain on the site, which was pushing my server’s CPU limits far more often than I would like. I found an up-to-date list of bad IPs from AbuseIPD maintained in a GitHub repository and dumped them into my Cloudron firewall (I noted that all of the recent malicious bots that had hit NLJ that day were included on the list). Did it have an effect? I present the following chart showing CPU and RAM usage for NLJ over the previous seven days.

Screenshot from Cloudron UI showing 7-day CPU and RAM usage for The New Leaf Journal from about noon on 01/23/26 through the same time on 01/30/26. There is a noticeable huge dropoff on the CPU and RAM charts beginning in the middle of 01/29/26.

What do you think? Thoughts? As an aside, I will note that Koko Analytics, the free and open source local page visit counting solution I use here and on NLJ, shows no noticeable drop in legitimate visitor traffic since I started using Cloudron’s firewall yesterday. I am crossing my fingers that keeping the Cloudron firewall up to date will resolve the NLJ issues I have been having.

Posted on
by

From Macy McCue Bell in The Washington Times this morning (link):

Many internet users woke up Tuesday morning and logged onto social media, video games or news sites, only to see an error message. An issue with the web infrastructure company Cloudflare is causing rippling effects across multiple websites and social media platforms.

I can guarantee that no human visitors to this site or The New Leaf Journal saw a Cloudflare error message (or any Cloudflare message for that matter). Moreover, they would have been able to save links in Wallabag and not been thwarted by CAPTCHAs. My .htaccess rules may have led to the blocking of some bots and requests, however.

Posted on
by

I read an interesting Hacker News comment by user dumbfounder (great username!) on Hacker News today. I excerpt the pertinent part below:

I created a search engine that crawled the web way back in 2003. I used a proper user agent that included my email address. I got SO many angry emails about my crawler, which played as nice as I was able to make it play. Which was pretty nice I believe. If it’s not Google people didn’t want it. That’s a good way to prevent anyone from ever competing with Google.

dumbfounder

I have had problems with bad crawlers (especially bad AI cralwers) on my sites. At the same time, dumbfounder highlights the reverse side of the coin. Many sites block good crawlers such as robots.txt-respecting crawlers for indepdent search engines. While all webmasters are free to control access to their sites as they see fit, allowing Google and other select big tech search crawlers while excluding small and independent search crawlers both limits search diversity and prevents people who may rely on small or niche search engines such as Mojeek, Marginalia, or Seznam from discovering potentially interesting writing. I previously published an article on this issue advocating for webmasters who want to support an open web and search engine diversity to ensure that good crawlers from independent search engines and directories can access their sites.

Posted on
by
Filed an Issue Koko Analytics Recording SBInstitutionsBot/0.1 Requests as Visitors/Page Visits · Issue #296 · ibericode/koko-analytics by nafnlj (GitHub)
My Set-Up I run the non-pro version of Koko Analytics on my WordPress site. I disable cookies and exclude visits from all logged-in users. What Happened? On two separate days, Koko Analytics was sh...

On March `12 and 19, I received a highly unusual number of visitors and page views on The New Leaf Journal according to Koko Analytics. By “unusual,” I mean about 10X as many visitors as expected. No individual article had an unusual number of views and my Koko Analytics stats showed the usual list of referrers (mostly search engines) at usual referral numbers. I checked by server logs on the 19th and determined that the likely culprit was SBInstitutionsBot/0.1 (it appears to be a Japanese web crawler for AI), which was hammering my site with requests as I was seeing unusually high visitor numbers. I used Jeff Starr’s BBQ Pro firewall plugin to block the bot. My Koko Analytics stats quickly returned to normal. I reported the issue to Koko Analytics on GitHub, so I hope the relevant IP addresses are added to the do not count list if it turns out that my diagnosis was correct. (In any event, one of my former visitors, PoorlyConfiguredWebCrawler, was much better behaved than SBInstitutionsBot.)