Feature suggestion: Why not line-based (or even JSON) log format?

Posts: 6 · 3FileSync 24 Dec 2022, 12:09

Wouldn't it be a nice idea to have FFS create log files in a format like this...

{"level":"info","datetime": "<some timestamp>","job":"<name of my ffs job",<and so on ...>}

Why I am yearning for this?
I am an enthusiastic adorer of FreeFileSync, have been using it for years now in my home network to sync between various computers and the cloud, mainly for backup.
Now I am in the middle of starting to use Grafana (https://grafana.com) and Promtail (https://grafana.com/docs/loki/latest/clients/promtail) for better tracking the status of my multiple automated FFS sync processes.

However, this is quite hard to achieve with current log file semantics:
Right now, FFS produces "multiline" log files, which are - strictly speaking -, rather "reports" than logs: In order to grab to relevant information from the logs, one has to read & understand each log file as a whole.
This is independent of the "log file format" I choose (plain text or HTML). Both results are semantically the same.
Such a thing is very hard to process automatically. Even though grafana comes with a specific "multiline" stage for processing "multiline" input (https://grafana.com/docs/loki/latest/clients/promtail/stages/multiline/), it is still unnecessarily complex, and in the specific case of my FFS logs, it's nearly impossible to achieve what I am trying to do.

I believe that FFS could be even more fantastic to use (than it already is) by integrating the option of a more contemporary log file output. This would ideally be a JSON structure.
Each line of the existing logs could be produced separately, with the main difference that each line would contain the same meta data like level, timestamp, job, status, total number of processed files/errors etc.

What do others think about it?

Posts: 1038 · therube 26 Dec 2022, 17:14

I've never liked json.
To me it is an unstructured "blob", potentially highly complex at that.
So unless you have something to parse that file...
And while there are json "prettifier's", that /display/ the code in a more readable manner, they do not expose the underlying context.

To me, if you want to parse the report, have the ability to output it in some sort of structured, single line per "record" text format, be it .csv or whatever.

A sample real .json.
And look at all the [REALLY] /diverse/ crap it (happens to) contains.
And being .json, this a single line of data (though here, it is "wrapped" in the code window).
Now for a "normal" human, it's tough to digest that.
For a computer, no big deal.
So when I want to pull "data" out of there, it's like, good luck.
Physically finding data, & understanding what you've found - in context, are totally different.

{"environment":{"locale":"en-US","localeLanguageCode":"en","browserSettings":{"update":{"channel":"nightly","enabled":true,"autoDownload":false,"background":false}},"attributionData":{},"currentDate":"2022-12-26T17:01:12.894Z","profileAgeCreated":1657125743342,"usesFirefoxSync":false,"isFxAEnabled":true,"isFxASignedIn":false,"sync":{"desktopDevices":0,"mobileDevices":0,"totalDevices":0},"xpinstallEnabled":true,"addonsInfo":{"addons":{"FirefoxColor@mozilla.com":{"version":"2.1.7","type":"extension","isSystem":false,"isWebExtension":true,"name":"Firefox Color","userDisabled":false,"installDate":"2022-07-21T19:03:19.338Z"},"{73a6fe31-595d-460b-a920-fcc0f8843232}":{"version":"11.4.13rc4","type":"extension","isSystem":false,"isWebExtension":true,"name":"NoScript","userDisabled":false,"installDate":"2022-07-06T16:47:15.837Z"},"addons-search-detection@mozilla.com":{"version":"2.0.0","type":"extension","isSystem":false,"isWebExtension":true,"name":"Add-ons Search Detection","userDisabled":false,"installDate":"2022-07-06T16:42:24.890Z"},"google@search.mozilla.org":{"version":"1.3","type":"extension","isSystem":false,"isWebExtension":true,"name":"Google","userDisabled":false,"installDate":"2022-07-06T16:42:25.215Z"},"amazondotcom@search.mozilla.org":{"version":"1.4","type":"extension","isSystem":false,"isWebExtension":true,"name":"Amazon.com","userDisabled":false,"installDate":"2022-07-06T16:42:25.280Z"},"wikipedia@search.mozilla.org":{"version":"1.2","type":"extension","isSystem":false,"isWebExtension":true,"name":"Wikipedia (en)","userDisabled":false,"installDate":"2022-07-06T16:42:25.429Z"},"bing@search.mozilla.org":{"version":"1.4","type":"extension","isSystem":false,"isWebExtension":true,"name":"Bing","userDisabled":false,"installDate":"2022-07-06T16:42:25.480Z"},"ddg@search.mozilla.org":{"version":"1.2","type":"extension","isSystem":false,"isWebExtension":true,"name":"DuckDuckGo","userDisabled":false,"installDate":"2022-07-06T16:42:25.499Z"},"ebay@search.mozilla.org":{"version":"1.4","type":"extension","isSystem":false,"isWebExtension":true,"name":"eBay","userDisabled":false,"installDate":"2022-07-06T16:42:26.023Z"},"screenshots@mozilla.org":{"version":"39.0.1","type":"extension","isSystem":true,"isWebExtension":true,"name":"Firefox Screenshots","userDisabled":false,"installDate":"2022-07-05T13:00:24.000Z"},"webcompat-reporter@mozilla.org":{"version":"1.5.1","type":"extension","isSystem":true,"isWebExtension":true,"name":"WebCompat Reporter","userDisabled":false,"installDate":"2022-07-05T13:00:24.000Z"},"formautofill@mozilla.org":{"version":"1.0.1","type":"extension","isSystem":true,"isWebExtension":true,"name":"Form Autofill","userDisabled":false,"installDate":"2022-07-05T13:00:24.000Z"},"pictureinpicture@mozilla.org":{"version":"1.0.0","type":"extension","isSystem":true,"isWebExtension":true,"name":"Picture-In-Picture","userDisabled":false,"installDate":"2022-07-05T13:00:24.000Z"},"webcompat@mozilla.org":{"version":"109.4.0","type":"extension","isSystem":true,"isWebExtension":true,"name":"Web Compatibility Interventions","userDisabled":false,"installDate":"2022-07-05T13:00:24.000Z"}},"isFullData":true},"searchEngines":{"current":"google-b-1-d","installed":["google-b-1-d","amazondotcom-us","bing","ddg","ebay","wikipedia"]},"isDefaultBrowser":false,"devToolsOpenedCount":5,"topFrecentSites":[{"url":"https://www.safeway.com/","host":"www.safeway.com","frecency":32744,"lastVisitDate":1671814773183},{"url":"http://forums.mozillazine.org/","host":"forums.mozillazine.org","frecency":19656,"lastVisitDate":1671734835399},{"url":"https://www.lidl.com/","host":"www.lidl.com","frecency":10667,"lastVisitDate":1669923072750},{"url":"https://shop.aldi.us/store/aldi/search/milk","host":"shop.aldi.us","frecency":6696,"lastVisitDate":1671049094981},{"url":"https://translate.google.com/","host":"translate.google.com","frecency":5544,"lastVisitDate":1668800141609},{"url":"https://bugzilla.mozilla.org/","host":"bugzilla.mozilla.org","frecency":5043,"lastVisitDate":1668546012804},{"url":"https://www.bjs.com/product/tyson-frozen-all-natural-white-meat-chicken-nuggets-5-lbs/3000000000000943632","host":"www.bjs.com","frecency":4729,"lastVisitDate":1671652186881},{"url":"https://postimages.org/","host":"postimages.org","frecency":4719,"lastVisitDate":1669221444805},{"url":"https://encode.su/threads/1707-CHK-Hash-Tool/page8?highlight=chk","host":"encode.su","frecency":4512,"lastVisitDate":1669234322694},{"url":"https://freefilesync.org/forum/viewforum.php?f=8","host":"freefilesync.org","frecency":4376,"lastVisitDate":1671727473843},{"url":"https://google.com","host":"google.com","frecency":3860,"lastVisitDate":1669409104354},{"url":"https://support.mozilla.org/en-US/products/firefox?as=u&utm_source=inproduct","host":"support.mozilla.org","frecency":3653,"lastVisitDate":1663253144393},{"url":"https://fastcopy.jp/","host":"fastcopy.jp","frecency":3516,"lastVisitDate":1669751272637},{"url":"https://crowdbunker.com/v/FNMZhrXStc","host":"crowdbunker.com","frecency":3296,"lastVisitDate":1666727951673},{"url":"https://www.uwe-sieber.de/","host":"www.uwe-sieber.de","frecency":2947,"lastVisitDate":1669751288776},{"url":"https://www.voidtools.com/","host":"www.voidtools.com","frecency":2805,"lastVisitDate":1669837979776},{"url":"https://www.bge.com/Pages/default.aspx","host":"www.bge.com","frecency":2454,"lastVisitDate":1669927882539},{"url":"https://avidemux.sourceforge.net/","host":"avidemux.sourceforge.net","frecency":2105,"lastVisitDate":1669751383200},{"url":"https://www.allsync.biz/","host":"www.allsync.biz","frecency":2071,"lastVisitDate":1667839151282},{"url":"https://imgur.com/","host":"imgur.com","frecency":1999,"lastVisitDate":1655130330779},{"url":"https://www.fedex.com/fedextrack/?duplicateTrackingId=012345604039519&trknbr=012345604039519","host":"www.fedex.com","frecency":1870,"lastVisitDate":1667235024096},{"url":"https://www6.zippyshare.com/v/ukhyV3Ot/file.html","host":"www6.zippyshare.com","frecency":1853,"lastVisitDate":1671226939808},{"url":"https://www.mirrored.to/files/0ES7R6LV/MPK.rar_links","host":"www.mirrored.to","frecency":1853,"lastVisitDate":1671226908953},{"url":"https://www.videohelp.com/software/GridPlayer","host":"www.videohelp.com","frecency":1715,"lastVisitDate":1667937279867},{"url":"https://www.wg9s.com/comm-253/","host":"www.wg9s.com","frecency":1699,"lastVisitDate":1666883842993}],"recentBookmarks":[],"pinnedSites":[{"url":"https://amazon.com","host":"amazon.com","searchTopSite":true}],"providerCohorts":{"onboarding":"","cfr":"","message-groups":"","messaging-experiments":"","snippets":"","whats-new-panel":""},"totalBookmarksCount":11,"firefoxVersion":110,"region":"US","needsUpdate":false,"hasPinnedTabs":false,"hasAccessedFxAPanel":false,"isWhatsNewPanelEnabled":true,"userPrefs":{"cfrFeatures":false,"cfrAddons":false,"snippets":false},"totalBlockedCount":5558,"blockedCountByType":{"trackerCount":0,"cookieCount":1100,"cryptominerCount":0,"fingerprinterCount":0,"socialCount":13},"attachedFxAOAuthClients":[],"platformName":"win","isChinaRepack":false,"userId":"14816208-6737-4613-90ae-47ed4df21614","profileRestartCount":335,"homePageSettings":{"isWebExt":false,"isCustomUrl":true,"urls":[{"url":"chrome://browser/content/blanktab.html","host":"browser"}],"isDefault":false,"isLocked":false},"newtabSettings":{"isWebExt":false,"isCustomUrl":false,"isDefault":true,"url":"about:newtab","host":""},"isFissionExperimentEnabled":false,"activeNotifications":false,"isMajorUpgrade":false,"hasActiveEnterprisePolicies":true,"userMonthlyActivity":[[179,"2022-11-29"],[110,"2022-11-30"],[160,"2022-12-01"],[94,"2022-12-02"],[217,"2022-12-05"],[107,"2022-12-06"],[100,"2022-12-07"],[59,"2022-12-08"],[102,"2022-12-09"],[69,"2022-12-12"],[119,"2022-12-13"],[105,"2022-12-14"],[43,"2022-12-15"],[148,"2022-12-16"],[43,"2022-12-19"],[29,"2022-12-20"],[152,"2022-12-21"],[130,"2022-12-22"],[93,"2022-12-23"],[1,"2022-12-26"]],"doesAppNeedPin":true,"doesAppNeedPrivatePin":true,"isBackgroundTaskMode":false,"backgroundTaskName":null,"userPrefersReducedMotion":true,"colorwaysActive":true,"userEnabledActiveColorway":false,"inMr2022Holdback":false,"distributionId":"","fxViewButtonAreaType":null},"version":1}

Now, if you're able to simplify things such that an output from FFS to .json yields a relatively easily human readible file, that'd be fine (but I'd still perfer plain text).

(Funny all the things you might infer from that .json file, heh.
And that is just what "tracking" companies do all the time.)

Posts: 6 · 3FileSync 26 Dec 2022, 18:45

@therube:
Of course I do not mean to "insist" on JSON as the way to eternal happiness ;-). I just consider it "ideal" from my perspective, as e.g. Grafana (which I am using, and it is probably one of the most famous monitoring solutions out there), and also Promtail, offer easy-to-use ways for parsing JSON.
In my opinion, the strength of JSON does NOT lie in human-readability, but in easy automation. And it is widely spread.
Or, in other terms: JSON spares me the tedious regexping. But JSON is not the only way around, of course.

I could also live quite well with what you are suggesting by "the ability to output it in some sort of structured, single line per "record" text format, be it .csv or whatever".

For instance, the following can also easily be parsed by Grafana, AND it is human-readable
(taken from my local Acronis True Image log files):

<some timestamp><BLANK><some run id><BLANK><some msg id><BLANK><a text message><eol>

This would leave my happy, too... if FFS would produce such a pattern.