Monthly Archives: November 2024

How is my new journal setup working for me?

by

I’ve started following more pen/planner blogs and a few of them mentioned the concept of “Techo Kaigi” which apparently translates roughly to “Planner meeting” and the idea seems to be that you take some time to evaluate how a system is working for you. I’ve seen people doing it more as an annual review, but since I’m using small notebooks and mine’s 2/3 done I think now’s as good a time as any to reflect on how the “new” setup is working for me.

All the things in my journal/planner set up laid out: two pouches, a Travler's notebook calendar with a zipper pouch attached to the back, a pencil + lead + eraser, a blue A5 notebook, and 6 fountain pens.
Image description: All the things in my journal/planner set up laid out: two pouches, a Travler’s notebook calendar with a zipper pouch attached to the back, a pencil + lead + eraser, a blue A5 notebook, and 6 fountain pens.

Current setup:

  • A5 dot journal (nominally a bullet journal but at this point I’m mostly using my own personal system)
  • Monthly calendar Traveler’s Notebook standard size (A5 slim)
  • Traveler’s Notebook zipper pocket for storing stickers (attached to calendar)
  • Fabric Zipper pouches: one for notebooks, one for pens
  • ~6 fountain pens
  • Pencil with my name lasered onto the side, eraser (in a penguin-shaped case) and a box of pencil lead

Previous posts talked about choosing the bullet journal itself (A Rhodia sewn spine softcover) and also switching to use a smaller calendar (Traveler’s notebook Monthly).

Calendar

The Traveler’s Notebook calendar turned out to be a great choice, and despite my worries it seems to be (just barely) big enough for me. I really love it with the zipper pouch attached for sticker management. I was worried that the thinner paper might bug me, but it’s working ok with the pencil I use since I tend to move things on the calendar sometimes. I’m getting into the habit of using washi since a lot of my stickers are too big to really fit in there. I suspect the calendar is going to really shine as I start to swap the bullet journal notebooks out more quickly for the next while. But it’s already been handy for an overview of school and kid related stuff!

The inside of my calendar notebook, showing a few upcoming days and some cute hedgehog stickers I got at Powell's.
Image Description: The inside of my calendar notebook, showing a few upcoming days and some cute hedgehog stickers I got at Powell’s.

I’ve been using my remaining monthly calendar stickers to make smaller spreads in the bullet journal because they’re too big for the Traveler’s notebook. But I don’t really *need* those calendar spreads in two places and I don’t want to keep more than one paper calendar updated. I’m debating some sort of art page as a month section break, or maybe this is time for some of my bigger stickers to shine?

Thinner A5 Journal

The smaller size of my new tiny softcover sewn-spine Rhodia notebook has meant that I carry the journal around a lot more than my old corgi journal. It lives in my knitting bag and even came on my last trip. So the smaller size has worked exactly as I hoped: big success!

But it’s maybe a bit more of a success than I was planned for: between the fact that the book is always close at hand *and* my new collection of fountain pens that makes it more fun to write, I’m filling this up faster than I realized I would. The notebook is only going to last 2 months instead of the 5 I estimated when I bought it many months ago before finishing my old journal. And that’s even though I moved a lot of tracking into the calendar notebook! If I add in stuff like a daily drawing challenge I’m quickly going to wind up with 1 notebook per month.

A stack of notebooks in different thicknesses. On the bottom is my original corgi journal that lasted two years which is the thickest of the 4 notebooks. Above that is my current 2-month journal which is the thinnest. On top are two more notebooks both about twice as thick as my current book. The red one says "clairfontaine" on the spine and the teal one says "rhodia"
Image Description: A stack of notebooks in different thicknesses. On the bottom is my original corgi journal that lasted two years which is the thickest of the 4 notebooks. Above that is my current 2-month journal which is the thinnest. On top are two more notebooks both about twice as thick as my current book. The red one says “Clairfontaine” on the spine and the teal one says “Rhodia”

I don’t know if that bothers me that the notebooks won’t last too long. I like the convenience of the lighter weight little notebooks, and I’m not too sad to have an excuse to switch notebooks multiple times a year and get that “fresh start” feeling. I guess it’s more expensive, but not enough to be a problem for me.

I’ve already picked up a few similarly sized notebooks, and also a few that are around double the thickness. (Thank you sales; you can see a few of the thicker ones in the picture above.) It might be logical to swap between thin and thick so I never wind up carrying two thick ones, but I think given the success of this notebook, I’m going to plan for another thin one next and see how the switchover goes.

I picked up one of those Iroful books that have paper designed to show off fancy ink and I think it might be fun to use that one next in conjunction with the Diamine Inkvent calendar since I’ll be using new inks every day for most of the month. But I’m going to swatch some of my current pens in there first to see how the whole thing feels before I decide for sure. I did decide that I’m going to start swatching pens in the *back* of notebooks because then there’s space to grow.

Inks

I did have my first pen + ink + paper combo complete fail with Octopus Sheening in Fairy. It worked beautifully in my dip pen on the sheets I bought for doing swatches (A white Rhodia pad), but was a disaster in my TWSBI Eco on the Rhodia ivory paper and it bled through everything. It was even worse in Tomoe River S notebook I sometimes use for pen testing and scribbles when I don’t want to break up a journal entry. (It was nice on my old journal with the 120gsm paper, but that’s not very helpful since there are no blank pages left in there!) I wound up clearing out the Eco and put a tiny amount in a Platinum Preppy and the fine nib has made the Fairy usable, but I’m not really getting sheen. Thankfully it’s a really nice colour so I’m happy to use what little is in there, but I’ll probably try it again when I switch paper.

A pair of images side-by-side showing the front and back of a page in a Tomoe River S notebook. At the top you can see a bunch of scribbles in Wearingeul 1984 in a Nahvalur Original Plus, including some thick blobs. On the bottom is some Octopus Fairy ink also with some blobs and writing. On the reverse side of the page, you can see that the 1984 does not show through but the Fairy has bled right through the page in a lot of places.
Image Description: A pair of images side-by-side showing the front and back of a page in a Tomoe River S notebook. At the top you can see a bunch of scribbles in Wearingeul 1984 in a Nahvalur Original Plus, including some thick blobs. On the bottom is some Octopus Fairy ink also with some blobs and writing. On the reverse side of the page, you can see that the 1984 does not show through although it has made some wet waves on the page, but the Fairy has bled right through the paper in a lot of places.

Overall, I’ve learned that while I’m usually team sparkle, the shimmer inks tend to either unpleasant to use (my Robert Oster Rose Gold Antiqua sample *squeaked* on paper in my Eco and caused a lot of hand strain) or underwhelming with only occasional peeks of shimmer in the first few lines (such as Robert Oster Emerald of Chivor, which was also at the edge of bleedthrough sometimes, and Ferris Wheel Press Crystal Blue Legacy which is 90% boring with occasional spectacular blue). I did like the Wearingeul Frankenstein and 1984 even without much luck on the shimmer actually showing on the page, at least. I’m slowly learning which pens go best with which inks and how carefully and slowly I have to write for shimmer inks to get the best effect, but it’s a slow learning process. I don’t think I’m ready to give up on shimmer inks entirely, but I think the annoying factor is going to change how I plan to use shimmer inks — more ornamentation, less journalling, and maybe not too many inked at a time.

I’m not too worried about having a few lousy ink experiences, though. That’s the point of trying samples! But also, most of these aren’t going on my list of full-sized bottles to buy.

One sparkle success story, though: I tried the Diamine Red Lustre that I hated in my Metropolitan again in November, this time in the TWSBI Swipe. I did have to dilute the ink a bit and I still don’t think it’s a nice journalling pen, but it’s been fun for section headers and drawings.

A drawing of a cartoon polar bear holding a heart. All the inks used have shimmer, but the gold shimmer on the red heart stands out particularly well.
Image Description: A drawing of a cartoon polar bear holding a heart. All the inks used have shimmer, but the gold shimmer on the red heart stands out particularly well.

And I have been enjoying a lot of inks, though! I finally swatched all my samples (maybe more on that in a future post) and I’m down to only 6 that haven’t made it into my journal rotation. At this point those will likely wait until January or later, since I’m going to be playing with inkvent inks in December.

Stickers

I picked up a Halloween countdown from Stickii and have been having a lot of fun using those stickers! I also dug out some other stickers I had around and have been using them in the journal. I do think it means I tend not to draw as much on my own when I have art to just paste in, but the stickers delight me regularly and it’s nice to have art *especially* when I’ve been making my hands sore from ink experiments and I wasn’t going to doodle with a pen/ink combo that’s making me cranky. Plus, it’s nice to have a relatively inexpensive way to support artists without winding up with piles of prints building up in my house. I have a substantial box of prints I don’t even have space to display, so it’s nice to use up sticker sheets every few weeks.

I’m debating getting a regular sticker subscription from stickii for my birthday, but I might wait until spring since I’ve got their advent binder to open in December and I’m definitely not going to finish all of that in one month!

Zipper pouch attached to to traveler's notebook calendar. This shows the front of the calendar with the zipper pouch sticking out to one side. There's a sticker from BSides PDX featuring a sasquatch holding a jack-o-lantern, and stamp-shaped one from Oblation Press with a dog on it in the zippered pouch.
Image Description: Zipper pouch attached to to traveler’s notebook calendar. This shows the front of the calendar with the zipper pouch sticking out to one side. There’s a sticker from BSides PDX featuring a sasquatch holding a jack-o-lantern, and stamp-shaped one from Oblation Press with a dog in fall scene on it in the zippered pouch. On the front cover of the calendar there is a big round shiny sticker with an aurora over mountains, and a smaller sticker with a orange hat wearing a witch’s hat that reads “today is a good day to get cozy”

Sticker storage was a bit of an issue because sometimes they got a bit rumpled from me pulling the other notebooks in and out of my pouch, but I got a Traveler’s notebook add-on that I’ve slipped over the back cover of the notebook that gives me a couple of pockets that are the right size for the sheets I have from a few different people.

View of zipper pouch attachment on back of my notebook, flipped "open" so you can see that some loose stickers are in the pouch and sheets are held in a pocket against the back cover.
Image Description: View of zipper pouch attachment on back of my notebook, flipped “open” so you can see that some loose stickers are in the pouch and sheets are held in a pocket against the back cover.

Pens

I now officially have “enough” pens for my usual needs: I wanted 4-6 for journalling, 2 for my backpack, and I added a couple to my desk for work todo lists. I used to do the work todo stuff digitally but it wasn’t working well so I decided to go analog to help myself break out of a rut. So far it’s helping!

From starting in May with my 1 wood pen, I’ve acquired about 2 more per month so I’ve amassed more than a dozen pens. Most of these are under $30 (often a lot less) so they’re in that “I don’t really have to think too hard about this purchase” level of things for me. I’ve tried to focus on trying different brands and different nibs and making sure I think about the ergonomics and use them a fair bit before letting my feelings about them really gel.

Thoughts on nibs:

  • Not a fan of Fine or Extra Fine for long-form writing, but being able to use them on cheap old notebooks is kind of great so they’re still useful to me.
  • Medium is convenient for maintaining some form of legibility when I want to write a little faster. It’s often my go-to on nights where I only have maybe 5-10 minutes to journal and don’t want to think about how I write.
  • I don’t own any Broad or extra/double broads myself, but I tried a few in store and decided they weren’t as much fun as stub nibs or as convenient as mediums. Maybe I’ll get some and change my mind eventually but it didn’t seem worth prioritizing.
  • I’m still loving stub nibs: the line variation is fun, they force me to write big, and as long as I’m a bit careful about my in choices of ink and how I write they can be pretty smooth.
  • I only just got a flex nib and have written with it twice, but it seems nice? I think the pen is too heavy for me though.
  • I really liked the fude nib on my dip pen, so I may have to invest in a regular pen that has one.

Thoughts on pen aesthetics:

  • Other people seem to care a lot about clips but I don’t think I’d miss them with my current setup. In fact I think the clips might be what scratched up one of my smaller plastic pens!
  • I do love sparkle on the outside even if I have mixed feelings about it on the inside.
  • I really like having at least a small window to view ink. (Especially the sparkly inks!)
  • I can handle much heavier pens than I might have guessed. Only one of my pens seems to be too heavy for longer use and I was well-warned about it (but decided to try it anyhow because it was on sale).
  • I do like the light weight ones, though! I was worried because I saw people talking about pens feeling “cheap” but so far only one of my plastic pens feels not great to me and it’s far from the cheapest of the lot.

Thoughts on filling mechanisms:

  • I often switch ink at the end of the month before pens would naturally run out of ink, so huge reservoirs aren’t super important to me right now. They might matter more when I’m not operating mostly off samples, but I suspect not because picking palettes for each month is something I really enjoy *and* because it’s good to be in a habit of cleaning the pens monthly.
  • It’s really convenient to use a syringe + converter to use the last of any sample vial rather than tryign to use a piston pen.
  • The converter pens are also pretty fast to clean compared to the piston ones. But my kid enjoyed cleaning my piston pen anyhow.
  • I haven’t tried to clean my one vacuum pen so no thoughts there yet. I’m intending to run it right out of ink which may take a while even though I tried not to fill it too much.
  • It takes me forever to empty a cartridge (in part because they’re such boring colours) so I haven’t tried refilling those yet.

I think it’s safe to say that I have an actual *collection* of fountain pens now. It’s not just the 6 you see but the other 8? or so scattered around my house. Maybe I could have saved some money by testing more pens in store, but I wouldn’t feel as confident about my choices if I hadn’t forced myself to use each pen in rotation for a month before moving on. And tester pens don’t tell you much about filling mechanisms, which I wanted to learn too. I’ve covered a lot of the things I wanted to try and I’ll probably give away a few of the pens that don’t suit me well as I replace them with ones that suit me better. I do think I’ll buy more pens: they’re smaller than yarn! But I think have a reasonable variety now and that’ll be perfect for experimenting with inks in December. And maybe I’m at the tipping point where I’m ready to be more picky about my choices which may help me resist overdoing it in the sales to come! (Well, one can hope.)

Bags

I remain a ridiculous Tom Bihn devotee and like being able to clip this whole thing into my knitting bag for easy retrieval. I spent a lot of time looking at notebook covers seeing if I could find something I’d like better than the A5 pouch and so far the answer is no. This cover has protected my setup really well and I’m really happy with how it worked out.

The small pouch works well as a pencil case, but I did notice that one of my smaller pens did get a tiny bit scratched up and the position of the scratch makes me think that it came from another pen’s clip. It’s not a big deal, but I will probably use this as an excuse to shop for pretty fountain-pen padded cases (or make my own). For now the one pen that’s prone to scratching has been moved to another pocket of my knitting bag but I may make a tiny sleeve for it so I don’t take up brain space thinking about it.

I am debating adding a second small pouch so I can have scissors and washi tape on hand too, but sometimes the washi tape gets kind of banged up if I carry it around. Since I usually only want those things at the beginning of the month when I’m setting stuff up, it’s just as well to have them live elsewhere in the house, but maybe I’ll find a tin of the right size in my knitting stash to solve the problem.

Overall

The pouches + notebooks + pens setup is working better for me than the larger planner in an organizer bag was. The new pouch comes around the house with my knitting, it’s easy to grab and throw into my suitcase, or even just to take out and put on my lap so I have my usual tools at hand. I did have to add some sticker storage but otherwise it’s pretty much as I’d planned before I started using it.

We’ll be testing how I handle more rapid journal swaps sooner than I expected, but I’m excited to try more paper and the calendar should help with continuity, so hopefully that’ll be fun instead of annoying.

I’m really delighted with having such a tangible way to show how fountain pens are changing my habits in an enjoyable way. So much more writing and a bit more drawing! And I’m also happy to be having fun with stickers, which I’ve always enjoyed but there’s only so much room on my laptop and the like. It’s funny to think that when I started journaling, I was thinking a lot about doing a gratitude journal because my grandmother had been keeping one to help with her mental health. But even when I wind up using the journal to grump about work or whatever, I’m getting a lot of joy from the process of picking up a pen and making the ink flow. It’s been a grumpy couple of months and I’m glad to lean in to stuff that’s fun and low-key creative.

Shark Pen!

by

3

It’s a fountain pen shaped like a shark! It’s made by Jinhao, who are known for making cheap but often decent fountain pens. It sounds like not all of their pens are winners because quality control isn’t great but if you’re willing to roll the dice and don’t mind that the design may be a total knockoff, sometimes you get a pretty decent pen at a discount price.

A shark-inspired fountain pen sits on my desk with two kitty pencil sharpeners. The shark pen has a shark shaped head with eyes, gills and a small dorsal fin. There is no tail on the other end of the pen; it tapers to a slightly smaller cylinder. The pen is made of a silvery blue/grey plastic with a clear section in the middle so you can see the ink. This section is a bit thinner than the rest of the pen and has some dents to support a triangular grip.
Image description: A shark-inspired fountain pen sits on my desk with two kitty pencil sharpeners. The shark pen has a shark shaped head with eyes, gills and a small dorsal fin. There is no tail on the other end of the pen; it tapers to a slightly smaller cylinder. The pen is made of a silvery blue/grey plastic with a clear section in the middle so you can see the ink. This section is a bit thinner than the rest of the pen and has some dents to support a triangular grip.

My shark pen cost $4 and was an impulse add to hit free shipping or something, but you can get them considerably cheaper from Ali Express or Amazon — search for Jinhao 993 or Jinhao shark pen. If you buy a pack of them I think they’re less than $2 each, which is pretty sweet for a pen with an included converter. It’s a bit longer than most of my other pens. Here’s a photo showing it with the Platinum Preppy and Pilot Varsity, both similar pens appreciated for their cheap prices.

Jinhao Shark Pen, Platinum Preppy Wa, and Pilot varsity. The shark pen is longer than the preppy which is in turn longer than the varsity.
Image Description: Jinhao Shark Pen, Platinum Preppy Wa, and Pilot Varsity. The shark pen is longer than the Preppy which is in turn longer than the Varsity.

I’m impressed at how nicely it writes. It’s got a very fine tip, so it’s not ergonomically great for *me* but as long as I’m not writing pages of stuff it’s pretty decent for notes and todo lists, and still a bit easier on my hands than a ballpoint. It’s thin enough that it works nicely without show-through on my thin-paged calendar and on cheaper notebooks without fancy paper. Well, it doesn’t show through in normal use: my kid definitely managed to get it to bleed, but that was very intentional on his part as he was exploring how the pen worked.

Shark pen sitting on my Field Notes (larger size) notebook that I carry around. It has been filled with kid doodles while my kid was playing with the pen, including a stick dog which has been labelled "dog" a butteryfly, a rainbow, a happy face and more. Most has been drawn with the shark pen although he's added some accents in purple (using my Pilot Kakuno)
Image Description: Shark pen sitting on my Field Notes (larger size) notebook that I carry around. It has been filled with kid doodles while my kid was playing with the pen, including a stick dog which has been labelled “dog” a butteryfly, a rainbow, a happy face and more. Most has been drawn with the shark pen although he’s added some accents in purple (using my Pilot Kakuno)

I bought it with the intention of it being a fun pen to have in my backpack for kid entertainment, and I particularly appreciate that it’s got a bit of plastic covering most of the nib, which makes it considerably less messy to hand to my child. (I’m not sure all versions of the pen have this, but mine does.).

Shark pen unchapped on my book. You can see that there is a black "hood" over the fountain pen nib. It is sitting on a notebook where you can see that my kid was delighted to discover that if he held the pen at the right angle he could get the ink to bleed through. Hands on learning!
Image Description: Shark pen unchapped on my book. You can see that there is a black “hood” over the fountain pen nib. It is sitting on a notebook where you can see that my kid was delighted to discover that if he held the pen at the right angle he could get the ink to bleed through. Hands on learning!

It is worth $4 for me but I’m mildly regretting not shopping around and getting a set instead, especially since there’s a good chance my one pen will wind up meeting an ignoble end while providing child entertainment. Oh well, maybe I’ll get a set next time if that happens!

Jinhao Shark Pen in blue/grey. It's a pen with a shark head shaped cap.
Image Description: Jinhao Shark Pen in blue/grey. It’s a pen with a shark head shaped cap.

A trip to Oblation Papers

by

I had a little bit of solo time on my way home from BSidesPDX in October, so I stopped by Oblation Papers. I couldn’t remember if I’d ever visited before — maybe once when I was visiting before we moved to the area? Anyhow, it’s very pretty:

The inside of Oblation Papers, a stationary shop in Portland. There are calendars and notecards arranged on a table in front, a mobile made of white paper flowers (?) hanging from the ceiling, an ink bar barely visible on the right, and more products including wrapping paper visible in the room beyond.
Image Description: The inside of Oblation Papers, a stationary shop in Portland. There are calendars and notecards arranged on a table in front, a mobile made of white paper flowers (?) hanging from the ceiling, an ink bar barely visible on the right, and more products including wrapping paper visible in the room beyond.

I mostly went to look around, but I did have one intended purchase: I wanted to take a look at the Traveler’s Notebook line they had to see if I could find a nice way to hold my Stickii sticker sheets with one of their folder-y things. It was really nice to see the options in person. I decided to grab the zippered pouch and after a bit of experimentation have hooked it over the back cover of my calendar and stuffed the stickers inside like so:

A traveler's notebook reuglar size monthly planner with a "zipper pouch" slipped over the back cover and stickers slipped inside. The zipper pouch itself is empty but has a large knitting "progress keeper" shaped like a lollipop sitting on it to hold it open for a picture.
Image description: A Traveler’s Notebook regular size monthly planner with a “zipper pouch” slipped over the back cover and stickers slipped inside. The zipper pouch itself is empty but has a large knitting/crochet “progress keeper” shaped like a lollipop sitting on it to hold it open for a picture. There is a sticker sheet with magical cats and fountain pens (designed by Yudoart) sitting on the top of the small stack of stickers stuffed into the pocket.

I’m glad to report that it fits both the stickii halloween stickers I got and just barely fits the pipsticks ones I use for some tracking since the sheet they’re on is a bit wider. The new A5 bullet journal I switched to in October doesn’t have a pocket, so this is my new solution! I actually like it better than the pocket of my old journal because the stickers are visible which helps me remember to use them and also adds some fun to the back of my calendar without me having to commit to seeing the same stickers all year. Although I did finally choose a couple for the front!

A beige traveler's notebook monthly planner in the regular size. You can see the zippered edge of the pouch sticking out on one side. On the front there are two stickers: a sleepy orange cat with a witch's hat that reads "today is a good day for getting cozy" (from The Latest Kate) and a large circular sticker with mountains, stars, a moon, and a purple to blue aurora that practically glows due to the reflective nature of the sticker. It's even prettier in person, and was made by Tonkai / Fireside Textiles.
Image Description: A beige traveler’s notebook monthly planner in the regular size. You can see the zippered edge of the pouch sticking out on one side. On the front there are two stickers: a sleepy orange cat with a witch’s hat that reads “today is a good day for getting cozy” (from The Latest Kate) and a large circular sticker with mountains, stars, a moon, and a purple to blue aurora that practically glows due to the reflective nature of the sticker. It’s even prettier in person, and was made by Tonkai / Fireside Textiles.

Both of those stickers came from regular monthly clubs, but you can buy your own pretty aurora sticker if you want! (They are so pretty.)

I did debate getting an actual Traveler’s Notebook leather cover to go with my calendar and took some time to feel the ones they had on display and think about it. I *think* they’re a bit too heavy and thick for what I want at the moment. I love the idea so much that I might try it someday anyhow, but I have to be fairly careful about adding too much weight to what I carry on the regular, so I settled for the cover and a shop souvenir sticker instead.

I also took some time to try out some oft-recommended beginner pens that were on my potential to-buy list as well as whatever else they had out. It turns out that I don’t actually love the feel of the Lamy Safaris, which isn’t too disheartening since I also don’t love most of their designs. Plus, some weeks after the day I was shopping they went and announced their new pens in partnership with the transphobe fantasy marketing machine (aka, HP) so I’m not feeling bad about taking them off the shopping list.

I had more luck with the Kaweco Sport: I do indeed like the feel of the pen, and after experimenting with the testers they had out, I’ve decided that double broad and broad are probably a bit too much for me, especially if I wanted to use it as a pocket pen. I don’t know that it’ll replace my space pen, but it seems like a viable contender. I wasn’t up for paying full retail the day I tried them but I did keep an eye out and later snagged one during the Fountain Pen Day sales so I might have more to say about that after it gets here. I did try a few other pens but none of them stood out enough to buy one.

Another view inside Oblation Papers, this time showing a different table and a large display of cards near the front windows.
Image Description: Another view inside Oblation Papers, this time showing a different table with 2025 calendars and a large display of cards near the front windows.

I did debate getting a bottle of one of their 4 shop-exclusive inks, but it was pretty busy that evening so I talked myself out of getting someone to get a bottle for me while I was waiting to check out. I feel like I have very little ink left because my sample vials are mostly empty, but I’ve bought a Diamine inkvent calendar so I’m going to have more than I can use next month.

Oblation papers was a lovely place to visit. Unfortunately, it’s more than an hour on the train and their prices and shipping are more expensive than some of the places I shop online, so I don’t think I’m going to suddenly become a regular customer even though it’s “local” to me. I do think being able to look through their ink swatch book in person might be handy for some ink purchases, but a few dollars for an ink sample seems like something I’m much more likely to do than a 3hr shopping trip. But I could see myself going down for an event or stopping by on those rare occasions where I’m down near the Pearl on my own!

Mirroring known vulnerability data globally, for free.

by

If you just want the answer to “where do I find a reliable global mirror of NVD vulnerability data?” or “Where should I get a list of CVEs if NVD is down?” the answer is https://cveb.in/ . It is being mirrored on the same servers used for major open source projects so you’re probably already trusting them, and they should be fast and may be very close to you. Please go ahead and use it and let us know how it works for you!

I co-presentented a talk about this work at BSidesPDX on Saturday, October 26, 2024:

Often when I write about talks I’ve given, I try to kind of recreate them in blog posts to be a bit of a director’s cut were I add in a bit of extra material that didn’t make the talk but they’re pretty similar to what I said on stage. This time, though, since I didn’t give the second half of the talk and John and I have very different ways of telling a story, I’m just gonna tell a story in this blog post and maybe toss in a few slides. If you want to watch us both tell the story from our own perspectives, check out the video. Although we collaborate on a lot of stuff it’s surprisingly rare for us to share a stage.

Still here and not going with the recording? Okay, let me tell you a story…

The US government DDOSed itself

Once upon a time, not so long ago, the US government decided it wanted to raise the bar for software security in their supply chain, and they wrote up an executive order on cybersecurity explaining how they wanted suppliers to do better, including a section on not shipping software with known vulnerabilities. Many other groups followed suit with similar recommendations or requirements.

As a result, a lot of organizations’ security plans started to look a lot like this:

A diagram from my talk about NVD mirroring. The top of the slide is labelled "2024 Corporate Security Policy_final_FINAL.doc" (which is a joke about filenames for things that undergo a lot of revisions). There are then three columns. The first is labelled Step 1 and there is text in a red box that reads "Scan components for vulnerabilities." Step 2 has an orange box which contains the text, "???" and Step 3 has two green boxes, one of which says "EVERYTHING IS SECURE" and has a picture of a closed lock. The second reads, "$$ Profit $$"
Image Description: A diagram from my talk about NVD mirroring. The top of the slide is labelled “2024 Corporate Security Policy_final_FINAL.doc” (which is a joke about filenames for things that undergo a lot of revisions). There are then three columns. The first is labelled Step 1 and there is text in a red box that reads “Scan components for vulnerabilities.” Step 2 has an orange box which contains the text, “???” and Step 3 has two green boxes, one of which says “EVERYTHING IS SECURE” and has a picture of a closed lock. The second reads, “$$ Profit $$”

There is a lot to say about steps 2 and 3 here, but our problem starts at the beginning of Step 1. To scan for vulnerabilities, you need a list of software you’re providing (which is a whole talk in and of itself) and a list of known software vulnerabilities.

One of the biggest sources of vulnerability data actually comes from the US government: the NVD (National Vulnerability Database) provided by NIST (National Institute of Standards and Technology). It’s pretty great — they provide it fully free, publicly licensed. This is usually where you go to get information about CVEs (Common Vulnerabilities and Exposures).

But what do you think happens if every single US government supplier and indeed, many other software companies around the world, all try to grab this data at once? And more than that, many of them start enabling regular scanning so they’re grabbing it multiple times per day, or per hour?

A slide from my BSidesPDX 2024 talk which reads "Distributed Denial of Service" and has photo I took of some street signs near the train tracks. The relevant one is a large yellow caution sign that shows a person with a bike getting a wheel stuck in the train tracks and the rider is being launched off the bike over the tracks.
Image Description: A slide from my BSidesPDX 2024 talk which reads “Distributed Denial of Service” and has photo I took of some street signs near the train tracks. The relevant one is a large yellow caution sign that shows a person with a bike getting a wheel stuck in the train tracks and the rider is being launched off the bike over the tracks.

So, yeah, the US government kind of started a denial of service attack against its own agency. And in case that wasn’t bad enough, we started seeing headlines like “NIST Struggles with NVD Backlog as 93% of Flaws Remain Unanalyzed ” where the stories talked about funding cuts at NIST.

The fine folk at NIST have been doing a hard job with not enough resources and some really unfortunate timing, so they’d already been working on keeping things from being overwhelmed. They had introduced rate limits per IP address/API key to keep rogue scanning jobs from ruining things for everyone, and they had started providing an API that allowed people to get just the newest data instead of having to download things every time. Unfortunately, the API combined with rate limits was pretty slow so getting the full database the first time using the API was onerous when it worked at all. Several of my colleagues in the UK and in India had such long delays that they had to give up and bootstrap the “old” way to get started. And a lot of people were running their scanning within ephemeral containers and just didn’t cache the copy of the database at all so they wanted to get all the data fresh with each new scan. When neither the rate limits nor the API was enough to address demand longer-term, and with budget cuts on the horizon, NIST turned to looking for industry partnerships and additional funding.

It was clear that this wasn’t a problem that was getting solved quickly.

That sounds bad, but how is that YOUR problem, Terri?

Why did I care? I mean, obviously I’m a security professional and things that stand in the way of good security choices are a problem for me in general. But in this case, my work open source project involves building a vulnerability scanner called cve-bin-tool: https://pypi.org/project/cve-bin-tool . It’s a free, open source software vulnerability scanner for binary files, git repos, and SBOMs.

(Quick reminder: This is my personal blog and as such, all opinions here are my own and do not necessarily reflect those of my employer.)

In the course of developing software to scan for vulnerabilities, we’d gotten a front row seat to all of the NVD changes: we’d had to start using API Keys and explaining them to our users, we’d had to handle new timeout messages and do appropriate backoffs and rate limits, and we’d started getting reports from users that updates were slow or not working. Many users and contributors located outside of the US were experiencing extensive delays.

Following NVD best practice had been making our code more complex, our software harder to use, and our users unhappy. It’s hard enough to get software developers to care about vulnerabilities, and it was getting uncomfortably hard to do something that had previously been pretty easy to install and try. But while we supported other data sources with vulnerability data, NVD was still the biggest one and the one people wanted the most.

How do we make vulnerability data available to everyone?

We probably could have solved the problem for cve-bin-tool similar to how commercial entities have handled it: make our own copy, query that, keep it updated separately. They often add proprietary data (such as the missing triage of new vulnerabilities) and then sell access to that data as part of their solution. We were already keeping a local copy of the data in github so our CI jobs would quit timing out at inopportune moments. But my goal has long been to make software more secure for everyone. What if we thought bigger than one python application? What if I built a solution that would help the whole world?

A slide from my BSidesPDX 2024 talk. On one side, it reads "what if we helped the *world* get vulnerability data?" and on the other side it has a screenshot of a tumblr post. The first post is from writing-prompt-s and reads "In a game with no consequences, why are you still playing the 'Good' side?". The next post is from raphaeliscoolbutrude and says "Because being mean makes me feel bad." The final post is from user deflare and reads, "Because my no-consequences power fantasy is *being able to help everyone*"
Image description: A slide from my BSidesPDX 2024 talk. On one side, it reads “what if we helped the *world* get vulnerability data?” and on the other side it has a screenshot of a tumblr post. The first post is from writing-prompt-s and reads “In a game with no consequences, why are you still playing the ‘Good’ side?”. The next post is from raphaeliscoolbutrude and says “Because being mean makes me feel bad.” The final post is from user deflare and reads, “Because my no-consequences power fantasy is *being able to help everyone*”

It might have been easy to lay a lot of the blame on people using “ephemeral” continuous integration jobs. They typically grab a mostly empty linux image, install/update some software, download the thing they want to scan, download the vulnerability data, store a report somewhere, then throw the rest of the thing away to start fresh next time. If they just cached the data instead of grabbing it every single time, we wouldn’t be in this mess.

But we could learn from what they were doing too: it was perfectly viable for them to download entire software binaries every single time, and no one batted an eye at that. Why was it easier and faster to get the software than to get meta data about the software? The answer, of course, is that we weren’t all trying to download from a single underfunded government agency. But instead we were downloading from… a bunch of underfunded open source hippies? How was that working but the government servers weren’t?

I am old enough that I knew the answer. Open source had solved their distribution problem by asking people to store a “mirror” (a copy of all the files) on their own servers, then building infrastructure to help people find the one closest to them. It all happened long before anyone had coined the term “cloud service provider” and it had happened on shoestring budgets with people donating a bit of space in a server rack and a bit of bandwidth. A lot of early mirrors were in universities or small internet service providers who had an open source enthusiast on staff. Get enough of them, and suddenly everyone gets software and no one gets stuck with a giant bill or an overloaded server.

It looked like neither government nor industry was going to solve this problem on the timeline I wanted and maybe never on the global scale that would make my life easier. But I have access to resources that a government agency maybe doesn’t: I know where one of the world’s leading experts on open source mirroring lives. It’s in my house. Because I married him. As well as having years of experience in multiple roles, he’s actively involved in running one of the larger open source content distribution networks in the world. So I had access to exactly what I needed to help everyone. I walked upstairs and said, “Hey John, if I wanted to mirror the NVD data on the micro mirrors, could we do that?” and then we figured out how to make it happen.

FCIX Micro Mirrors

This is the point at which I handed the talk over to John. But here’s my truncated version of his half of the story.

John builds infrastructure the way I knit: compulsively and constantly. And when he’s not actually doing something with his infrastructure there’s a good chance he’s thinking about it or talking about it. He hosts people’s websites and emails and mastodon accounts, he accidentally got involved in founding a whole internet exchange, and he’s forever automating and building backends for things in the house that I really wish weren’t internet-enabled. (Look, I’m a security professional, I’m allergic to too much internet.)

One day his friend Kenneth decided it would be fun to run a software mirror for their internet exchange, and he roped John into it, and then into this hare brained idea of maybe running a lot of mirrors on cheap hardware. John had previously run kernel.org and the associated linux mirrors there, and he had done so on big beefy servers with big beefy bandwidth, so he was skeptical that this would work. Still, not only was it cheap to try and see, but thanks to some donations they didn’t even have to lay out much of their own money to get it going. And long story short: it turns out it works incredibly well.

The deal is that they build up these cheap “thin client” boxes with a hard drive in them that have a copy of the data and are managed remotely by John and Kenneth. Then they offer them up to free to data centres who are willing to provide power and internet. It’s kind of a fully managed appliance, so the data centre gets blazing fast downloads of open source software for their customers and anyone else “nearby” and Kenneth and John get a dot on their map and the knowledge that they’re helping distribute open source software. (Also they get to run globally load bearing infrastructure for funsies. Which it really is for them.)

Here’s my favourite picture: since one of our contributors is based in the UK, we turned on the UK-based mirrors first, and one of them is a data center in a box in a field:

A dark green utility box sitting in a beautiful field with yellow summer grass, green bushes, and green trees along the edges. There is a wedge of blue sky with clouds visible. One of the software mirrors is inside the green box.
Image Description: A dark green utility box sitting in a beautiful field with yellow summer grass, green bushes, and green trees along the edges. There is a wedge of blue sky with clouds visible. One of the software mirrors is inside the green box.

What’s been amazing is that this little network of devices is now a major powerhouse of linux mirroring. They estimate that they’re providing 90% of the bandwidth used for VLC, so if you’ve downloaded that or anything else they serve, there’s a good chance you’ve already used one of these mirrors and not known it. https://mirror.fcix.net/ if you want to see the list of projects. Kenneth is giving a talk at SeaGL in November if you want to hear more about the micro mirror story.

Serving the right data: files are better than APIs

The key to using these tiny servers is basically “linux people optimized sending files in order to make mirrors work.” And they did that quite a while ago so it’s really stable and fast now. You might think “oh, couldn’t you use bittorrent?” but that adds a lot of overhead. (That paper is older, but the numbers haven’t made it look more appealing in the time since then.)

If we want to go with what works, then, we can’t mirror the NVD API — that would require processing and these mirrors are not that smart. But it turns out… people didn’t really love the NVD API. It definitely filled a need for some folk, but when they tried to turn off the old file-based data so many people protested that the original deadline for removing the files got pushed out and pushed out. So we can probably guess that many users would like the files as much or better than the API, assuming they could get them faster and without rate limits.

So here’s what it looks like:

  1. We are running our own API crawler
  2. Generating json files compatible to the original ones
  3. Signing those files with possibly the sketchiest gpg key on the planet
  4. Mirroring these files to a worldwide CDN we created
  5. Literally solving the entire API / DDOS problem for… free?

Since cve-bin-tool has to speak API already, we can have cve-bin-tool output valid json files when needed. Although since NVD is still providing the json files at the time of this writing, we can (and do) get their files directly.

I should note that the technical implementation and testing in a live environment took a few months once we decided to do it. Much faster than waiting for funding!

Why should you trust us?

First: we are not affiliated with NIST, and they were not involved in any of this. Although I did email them so they knew who was behind it in case it came up and I got a nice email saying effectively they don’t officially endorse anything, which is fine. I want to joke that I’m the pirate radio of vuln data, but recall that the data is licensed public so there’s no piracy involved. Just fast and efficient transmission of perfectly allowed data.

So why should you trust some internet randos to get you vulnerability data? After all, the software security industry tries to tell you to stop downloading files served by random people on the internet! But these are the same servers that you’re probably using to get security updates, so… you probably already do trust them?

For a lot of the software on these mirrors, it’s a trust-but-verify solution where the packages are signed and package managers validate those so even if one of the data centres wanted to serve up malicious code, it wouldn’t get auto-installed unless they also compromised some build and signing servers. So you’re trusting not just the mirror, but the whole process to make sure the mirror serves up the right data.

If you’re going to build some similar verification into your tool that uses NVD data, you can verify our (sketchy) gpg signatures so you know it came from us, but you can also validate the data against NVD itself. For the json files they provide some metadata you can use. If we’re generating our own json (as we expect to do when they turn off theirs) then it might get a bit more complicated, but you can probably figure something out. For example, if validating all the data is impractical, you could have something that uses the API to double-check only the CVEs you care about. You can also always use us as your “seed” source and then update against NVD directly thus overwriting as needed.

(Incidentally, don’t bother trying to run a json schema check on the data as part of your checks unless you like noise. We did this in cve-bin-tool and had to turn it to just warn instead of halting because NVD themselves produce invalid json files frequently enough that it was a problem. Turns out keeping a giant database full of user-submitted data valid is hard.)

Using the mirror

The instructions are here: https://cveb.in/

Basically, go nuts. Those little thin clients can handle full fedora releases and don’t even max out on release day any more. Please use them! They should be fast, they are probably significantly less overloaded than the main NVD servers, and there’s no rate limits or API keys needed. Plus, you’ll make pretty marks on John’s graphs.

You can also use the mirror data as part of cve-bin-tool so you don’t have to build your own scanning service!

Conclusion

I noticed a problem where software vulnerability data about CVEs was getting harder and harder to access, and roped the fine folk of the FCIX Micro Mirror project into hosting a copy of this publicly available data on https://cveb.in/ which they are doing for free thanks to donations of time, money, and server rack space from a variety of folk. These mirrors are fast, available worldwide, not rate limited, and we would love it if you used them.

Contacting us

The comments for this post will turn off after a few weeks because I don’t feel like dealing with spam, feel free to hit me or John up with questions on the fediverse anytime! We’d also to love to hear how you use https://cveb.in/

Future work

I’m not actively working on mirroring anything else at the moment, but I *do* think it would be super cool if we could get the micro mirror system to help provide files for pypi / pip. So if you’ve got a lead there and global distribution of python packages sounds like a good idea, let us know! And if you’ve got any other way we could make the world a better place for free, that’s cool too.