Add ETW-based process events table for Windows #7821

zwass · 2022-11-17T17:25:49Z

This PR implements POTE (see Blueprint) and the new evented table process_etw_events.

Blueprint: #7826

This work done by @marcosd4h. I just opened the PR.

… Pushing initial version of the ETW Process Event Publisher and Subscriber

zwass

I only took a quick look so far (mostly at the schema).

Very glad to see the thorough comments on all the ETW "framework" header files! Thank you for doing that.

After we get this initial work merged into osquery core, it would be really helpful if you could write up some documentation in the osquery repo about how further ETW tables should be built using the framework.

zwass · 2022-11-17T17:26:52Z

specs/windows/etw_process_events.table

+table_name("etw_process_events")
+description("Windows process execution events.")
+schema([
+    Column("type", TEXT, "Event Type"),


It would help to see the possible values. Is this right?

Suggested change

Column("type", TEXT, "Event Type"),

Column("type", TEXT, "Event Type (ProcessStart, ProcessStop)"),

good point! I think having the possible types listed here will help

zwass · 2022-11-17T17:28:12Z

specs/windows/etw_process_events.table

+    Column("header_pid", BIGINT, "Process ID of the process reporting the event", hidden=True),
+    Column("process_sequence_number", BIGINT, "Process Sequence Number - Present only on ProcessStart events", hidden=True),
+    Column("parent_process_sequence_number", BIGINT, "Parent Process Sequence Number - Present only on ProcessStart events", hidden=True),


Are these columns hidden because they are mostly useful for dev/debuggign?

Is the process_sequence_number unique at some level? Would be useful to have a non-PID unique identifier for a process

A corner case scenario on Windows called "PID reuse" can happen when the same PIDs get reused during OS runtime lifecycle. This is something very unlikely that OS developers decided to address by including the concept of the "process sequence number" on the ETW fields. I'm including this data as a hidden value and exposing this information in the unlikely event this is needed.

@zwass yes, these could be useful for debugging scenarios

@defensivedepth yes, they are unique

@marcosd4h How unique? Sysmon generates a GUID for each new process creation (https://learn.microsoft.com/en-us/sysinternals/downloads/sysmon#event-id-1-process-creation) specifically because of the issues around PIDs.

I would recommend that we do not hide the sequence number columns. For security use cases / correlation, some type of GUID would be better than PIDs.

@marcosd4h Wanted to make sure my reply didn't get lost in all the other comments

Hey @defensivedepth, sorry for the late response! I totally miss your response.

This sequence value the code is exposing is generated by the kernel itself. The code is just grabbing it from the ETW event and exposing it to the table. In other words, the code does not control this value; it just relies on the kernel-generated value to be unique for the current os boot session.

libraries/cmake/source/krabsetw/config/krabs.cpp

directionless

No real comments, but I'm excited to see it!

directionless · 2022-11-17T17:49:03Z

specs/windows/etw_process_events.table

@@ -0,0 +1,29 @@
+table_name("etw_process_events")


(This isn't blocking)

Does it make sense to expose any of this in the existing process_events? I feel like we've never really had a good answer for how we handle that, and the migration between event subsytems. (process_events vs bpf_process_events vs es_process_events)

I guess maybe it makes more sense to let them all diverge, and people can write platform specific SQL?

We discussed this yesterday with @zwass and other folks during the Osquery office hours. There is still not a clear decision on how to move fwd. We most likely go ahead with this change and then explore options to converge es_process_events, bpf_process_events, and etw_process_events into a single table.

My takeaway from office hours was:

We should keep separate tables

Future work to explore a merged process_events

We should probably name them process_events_etw so like tables sort together

I missed office hours, apologies, just my 2c here. I'm not a huge fan of diverging tables, as it puts the data in different places, and it's not nearly as intuitive as there being just one process_events table. The diverging schemas was what we had originally shipped OS specific columns for I thought.
I'm not going to stand in the way of progress here, just wanted to voice my thoughts on this.

Hey @muffins, thanks for providing your input here. During office-hours meeting we decided to merge es_process_events, bpf_process_events, and etw_process_events into a single table. There has to be some work involved to decide how to merge the schemas through os-specific columns.

@marcosd4h thanks for the context! This is what I get for not coming to OH :)

https://hackmd.io/F9uUgvUkQuuyfrPoZbstYg?view#ETW-events-table--Process-events

I think we were more nuanced. IIRC the extended schema has some issues under the hood, and it's often hard to see how to combine things.

So we were mulling on the idea of having divergent tables, and also a single table with a simplified set of things. But we didn't think we should block here on that.

We did most think we should start naming them process_events_XXXX so they sorted together

@directionless @zwass I ended up naming the table process_etw_events. The main rationale for picking this name instead of process_events_XXXX is that evented tables must have the _events suffix at the end of the table name

specs/windows/etw_process_events.table

…tanding

mike-myers-tob · 2022-11-22T19:14:56Z

About the failing Windows test:

81: Test command: D:\a\osquery\osquery\w\install\cmake-3.21.4-windows-x86_64\bin\cmake.exe "-E" "env" "PYTHONPATH=D:/a/osquery/osquery/w/build/python_path" "C:/hostedtoolcache/windows/Python/3.11.0/x64/python3.exe" "-u" "test_release.py" "--build_dir=D:/a/osquery/osquery/w/build" "--verbose"
81: Test timeout computed to be: 300
81: test_linked_system_libraries (__main__.ReleaseTests.test_linked_system_libraries) ... FAIL
81: test_no_nonsystem_link (__main__.ReleaseTests.test_no_nonsystem_link) ... skipped 'Test for Darwin and Linux only'
81: 
81: ======================================================================
81: FAIL: test_linked_system_libraries (__main__.ReleaseTests.test_linked_system_libraries)
81: ----------------------------------------------------------------------
81: Traceback (most recent call last):
81:   File "D:\a\osquery\osquery\w\build\tools\tests\test_release.py", line 155, in test_linked_system_libraries
81:     self.fail(
81: AssertionError: Found these additional unwanted libraries linked:
81:     tdh.dll
81: 
81: ----------------------------------------------------------------------
81: Ran 2 tests in 0.065s
81: 
81: FAILED (failures=1, skipped=1)
81/82 Test #81: tools_tests_testrelease ..........................................***Failed    0.18 sec

…ries as tdh.dll is going to be loaded at runtime to get ETW working

lcostantino · 2022-11-23T18:38:52Z

Hi @zwass , excellent work!, BTW i had a POC for the same, but using a different provider (not the kernel one) with further modifications to share code between process_events and processes while getting information not present in ETW event.

In case you are interested to add some improvements or fields in a future PR, feel free to use some of those changes. (lcostantino#1)

marcosd4h · 2022-11-24T16:33:27Z

Hi @zwass , excellent work!, BTW i had a POC for the same, but using a different provider (not the kernel one) with further modifications to share code between process_events and processes while getting information not present in ETW event.

In case you are interested to add some improvements or fields in a future PR, feel free to use some of those changes. (lcostantino#1)

@lcostantino Flaco!! mann, it's really a small world, after all! I'm happy to see you here. Also, glad to see that you are thinking that adding ETW support to Osquery is the way to go. Would it be possible for you to code review this PR if you have a chance? I agree that there are things in your PoC that can be used to enrich the process events created on this PR. I also created a framework to add new ETW-based Event publishers, described in the blueprint [here]. (#7826). Your feedback on the blueprint will be greatly appreciated. Are you on the Osquery slack by chance?

marcosd4h · 2022-11-24T16:38:55Z

Small clarification here: @zwass created this PR by mistake while we were reviewing the initial specs for this new evented table. That's why there are confusing details in the PR description.

This PR introduces POTE (Programmable OS Tracing Engine) framework + a new windows evented table called etw_process_events which is built on top of POTE. The main purpose of this new evented table is to audit process create/termination details.

Having POTE in place will simplify the addition of future evented tables as POTE provides a simplified mechanism to create ETW-based Event publishers.

The PR blueprint is detailed here.

lcostantino · 2022-11-24T17:06:42Z

Hi @zwass , excellent work!, BTW i had a POC for the same, but using a different provider (not the kernel one) with further modifications to share code between process_events and processes while getting information not present in ETW event.
In case you are interested to add some improvements or fields in a future PR, feel free to use some of those changes. (lcostantino#1)

@lcostantino Flaco!! mann, it's really a small world, after all! I'm happy to see you here. Also, glad to see that you are thinking that adding ETW support to Osquery is the way to go. Would it be possible for you to code review this PR if you have a chance? I agree that there are things in your PoC that can be used to enrich the process events created on this PR. I also created a framework to add new ETW-based Event publishers, described in the blueprint [here]. (#7826). Your feedback on the blueprint will be greatly appreciated. Are you on the Osquery slack by chance?

Hey Marcos, didn't knew it was you!!! happy to see you, sure i will check the PR. (not in OSQUERY slack yet)

lcostantino · 2022-11-24T17:13:22Z

osquery/events/windows/etw/etw_concurrent_queue.h

+  T pop() {
+    std::unique_lock<std::mutex> lock(mutex_);
+    while (queue_.empty()) {
+      condition_.wait(lock);


would it be safer to use wait with predicate for empty? (spurious wake should also be covered with that)

good point! Thanks for bringing it up. I've just modified the wait call to use a predicate to avoid lost or spurious wakeups

Smjert · 2022-12-12T19:28:35Z

@marcosd4h for the CVEs of krabsetw, the CI script will ask you to insert all fields for the library, but I checked in the NVD database and there's no CPE for the library.
You might want to check this new page and section: https://osquery.readthedocs.io/en/latest/development/cve-scan/#libraries-without-a-cpe

I think I need to improve the very first message when the library is not found, so that it points to the instructions for adding a new library and handling special cases.

swannman · 2022-12-12T19:52:24Z

@marcosd4h for the CVEs of krabsetw, the CI script will ask you to insert all fields for the library, but I checked in the NVD database and there's no CPE for the library.

Good catch @Smjert - I'll start a thread with NIST per this guidance to get a CPE created for krabsetw. Thanks for pointing this out!

Smjert · 2022-12-12T20:11:28Z

Good catch @Smjert - I'll start a thread with NIST per this guidance to get a CPE created for krabsetw. Thanks for pointing this out!

Hello @swannman and thanks! That's definitely useful!

marcosd4h · 2022-12-12T20:18:09Z

@marcosd4h for the CVEs of krabsetw, the CI script will ask you to insert all fields for the library, but I checked in the NVD database and there's no CPE for the library. You might want to check this new page and section: https://osquery.readthedocs.io/en/latest/development/cve-scan/#libraries-without-a-cpe

I think I need to improve the very first message when the library is not found, so that it points to the instructions for adding a new library and handling special cases.

Thanks @Smjert, for pointing this out. I updated the branch to get the new CI scripts and updated the library manifest file. BTW, the version field was required and KrabsETW library does not have tagged released versions, so I ended up just using 0.0.0.0. Let me know if that works here.

Smjert · 2022-12-12T20:23:11Z

Thanks @Smjert, for pointing this out. I updated the branch to get the new CI scripts and updated the library manifest file. BTW, the version field was required and KrabsETW library does not have tagged released versions, so I ended up just using 0.0.0.0 here. Let me know if that works here.

Yeah the version is mostly for human readers when there's no CPE, but if there's no version it's fine too, the commit sha is actually what gets compared in the repo.

muffins

Presuming you've changed the items you've noted, this is largely looking good to me. As mentioned I'd really love to have another C++ human look through, esp for potential design improvements, but overall I think it's fine.

I'm happy to stamp if needed, but I'm going to let other folks have a chance to look through. If no one else steps up ping me and I'm happy to approve.

muffins · 2022-12-15T04:32:50Z

osquery/events/windows/etw/etw_kernel_session.cpp

+  case EtwProviderConfig::EtwKernelProviderType::File: {
+    // https://learn.microsoft.com/en-us/windows/win32/etw/fileio
+    kernelProvider = std::make_shared<krabs::kernel::file_io_provider>();
+  } break;


Nit: I think you want these breaks inside of the curlys, keep all the code for each case together inside.

That seems easier to read to me as well. Any reason not to do it?

good point; I've just fixed that.

osquery/events/windows/etw/etw_provider_config.h

muffins · 2022-12-15T04:58:17Z

specs/windows/etw_process_events.table

@@ -0,0 +1,29 @@
+table_name("etw_process_events")


@marcosd4h thanks for the context! This is what I get for not coming to OH :)

Stale review.

…n logic. Improving shutdown logic on the ETW sessions runners. Adding more tests to cover the shutdown scenario. Adding integration test.

marcosd4h · 2022-12-28T15:34:02Z

Hey @muffins, I hope you are doing great! I wanted to let you know that I've just pushed integration and events tests for the ETW logic. I've also included tests for the concurrent queue code. Adding tests was a great idea as it helped me to improve the publisher shutdown logic. Thanks for bringing this up! Also, sorry for the delay in getting the tests in place; I've been buried with high-priority work at FleetDM and didn't have the time the last few weeks to work on this.

muffins · 2023-01-07T04:44:44Z

@marcosd4h no stress on the time delays, it was also holidays so I wasn't looking at computers much! I'll try and take another look through everything over the next week or two. Hopefully we can get this merged up soon. Thanks a ton for bearing with us and writing tests, I know everyone will be pumped for the extra assurances 🙏 🥳 🧑‍🏭

marcosd4h · 2023-01-09T18:54:23Z

@marcosd4h no stress on the time delays, it was also holidays so I wasn't looking at computers much! I'll try and take another look through everything over the next week or two. Hopefully we can get this merged up soon. Thanks a ton for bearing with us and writing tests, I know everyone will be pumped for the extra assurances 🙏 🥳 🧑‍🏭

Thanks @muffins, I hope you had a great start of the year!! I'm looking forward to the upcoming review. I'm open to work on any change that might be needed to get this merged 💯 🥳 🥳 🥳

zwass

@marcosd4h this is mostly nits and some requests for clarification.

This is tricky to read as such a huge PR, but it definitely helps that you documented many of the class members and methods. Thanks!

zwass · 2022-12-30T17:41:40Z

osquery/events/windows/etw/etw_controller.cpp

+
+namespace osquery {
+
+//  Returns a reference to the single global EtwController instance


Suggested change

// Returns a reference to the single global EtwController instance

// Returns a reference to the single global EtwController instance

zwass · 2022-12-30T17:41:45Z

osquery/events/windows/etw/etw_controller.cpp

+  return instance;
+}
+
+//  New events get store in the post-processing queue


Suggested change

// New events get store in the post-processing queue

// New events get stored in the post-processing queue

zwass · 2022-12-30T17:53:28Z

osquery/events/windows/etw/etw_kernel_session.cpp

+  case EtwProviderConfig::EtwKernelProviderType::File: {
+    // https://learn.microsoft.com/en-us/windows/win32/etw/fileio
+    kernelProvider = std::make_shared<krabs::kernel::file_io_provider>();
+  } break;


That seems easier to read to me as well. Any reason not to do it?

zwass · 2022-12-30T17:56:06Z

osquery/events/windows/etw/etw_kernel_session.cpp

+void KernelEtwSessionRunnable::initKernelTraceSession(
+    const std::string& sessionName) {
+  if (sessionName.empty()) {
+    return;


zwass · 2022-12-30T17:56:10Z

osquery/events/windows/etw/etw_kernel_session.cpp

+void KernelEtwSessionRunnable::stopKernelTraceSession(
+    const std::string& sessionName) {
+  if (sessionName.empty()) {
+    return;


zwass · 2023-01-31T18:44:19Z

osquery/events/windows/etw/etw_publisher_processes.cpp

+void EtwPublisherProcesses::initializeHardVolumeConversions() {
+  const std::string validDriveLetters = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
+
+  for (auto& driveLetter : validDriveLetters) {


Comment says done but looks like change wasn't pushed?

zwass · 2023-01-31T18:46:20Z

osquery/events/windows/etw/etw_publisher_processes.cpp

+      processStartAggregationCache_.erase(it++);
+    } else {
+      ++it;
+    }


Is this the same? Feels easier to understand for me to have the iterator advance outside the conditional

Suggested change

processStartAggregationCache_.erase(it++);

} else {

++it;

}

processStartAggregationCache_.erase(it);

}

++it;

Thanks for the suggestion, I think the code is more readable now. I'm going to try out this change and see how it goes

zwass · 2023-01-31T18:48:12Z

osquery/events/windows/etw/etw_publisher_processes.cpp

+        strlen(userNameStr.data()) == 0 ||
+        strlen(userNameStr.data()) >= MAX_PATH ||
+        sidType == SID_NAME_USE::SidTypeInvalid) {
+      // Inserting empty username to avoid the lookup logic to be called again


I see this comment but I don't actually see the insert. Am I misunderstanding?

Maybe log here as well?

Good catch! I've just added the missing insert here. This helper is used to populate an attribute of the resulting event, I'd prefer not to log here

zwass · 2023-01-31T18:49:16Z

osquery/events/windows/etw/etw_publisher_processes.cpp

+  case TOKEN_ELEVATION_TYPE::TokenElevationTypeDefault: {
+    tokenInfo.assign("TokenElevationTypeDefault");
+  } break;
+
+  case TOKEN_ELEVATION_TYPE::TokenElevationTypeFull: {
+    tokenInfo.assign("TokenElevationTypeFull");
+  } break;
+
+  case TOKEN_ELEVATION_TYPE::TokenElevationTypeLimited: {
+    tokenInfo.assign("TokenElevationTypeLimited");
+  } break;


Let's use the same break in the curly braces syntax here please.

zwass · 2023-01-31T18:53:52Z

osquery/events/windows/etw/etw_user_session.cpp

+      (ULONG)sizeof(EVENT_TRACE_PROPERTIES);
+
+  /// Best effort to stop ongoing trace session
+  ControlTraceA(NULL,


Could be nice to log the return value

good point, just fixed this

zwass

@marcosd4h thank you for addressing the comments!

This is looking ready to merge to me. I can't approve it since I opened the PR even though Marcos wrote all the code.

directionless

A somewhat blind 👍, since @zwass would have, but PR authorship happened. Consider the magic co-authored-by silliness.

zwass · 2023-02-06T18:40:45Z

uhoh, GitHub wants to mark me as the author in the squash commit. Anyone know a way to resolve that? I didn't write a single line of this.

directionless · 2023-02-06T18:47:26Z

uhoh, GitHub wants to mark me as the author in the squash commit. Anyone know a way to resolve that? I didn't write a single line of this.

Yes, that's how it works. The best you can do is Co-authored-by: Marcos Oviedo <marcos@fleetdm.com>

Could probably close this, and have Marcos open anew

zwass · 2023-02-06T19:57:37Z

Maybe I can squash this locally into a commit with Marcos' name on it and then we can "rebase and merge" that?

Marcos is out on Vacation this week so I'd rather not bother him to open another PR, and it would be nice to have the commit associated with all the comments here.

directionless · 2023-02-07T03:16:41Z

Maybe I can squash this locally into a commit with Marcos' name on it and then we can "rebase and merge" that?

Marcos is out on Vacation this week so I'd rather not bother him to open another PR, and it would be nice to have the commit associated with all the comments here.

Hrm. That sounds a little messy? We can try, but I don't think we can roll it back.

zwass · 2023-02-07T04:40:52Z

I tried the things I could... adding Marcos as "co-author" and clarifying in the commit message seems to be the best we can do right now.

marcosd4h added 2 commits November 17, 2022 12:45

Pushing initial version of the Programmable OS Tracing Engine (POTE).…

d284921

… Pushing initial version of the ETW Process Event Publisher and Subscriber

Removing duplicated file that was push by mistake

6599497

zwass requested review from a team as code owners November 17, 2022 17:25

zwass commented Nov 17, 2022

View reviewed changes

directionless reviewed Nov 17, 2022

View reviewed changes

marcosd4h added 4 commits November 18, 2022 11:23

Adding support for event type handle through EtwProviderConfig

9448831

Renaming addEventType to addEventTypeToHandle to improve usage unders…

23c9825

…tanding

Adding support for datetime in etw_process_events table

370e503

Improved addProvider logic. Added PreProcessor safety wrapper.

814f45f

mike-myers-tob added feature virtual tables events Related to osquery's evented tables or eventing subsystem Windows labels Nov 22, 2022

mike-myers-tob changed the title ~~Feature/windows process events~~ Add ETW-based process events table for Windows Nov 22, 2022

mike-myers-tob mentioned this pull request Nov 22, 2022

Port process_events and file_events to Windows #2567

Open

marcosd4h added 2 commits November 22, 2022 20:04

Updating windows_expected_libraries array in test_linked_system_libra…

be57acb

…ries as tdh.dll is going to be loaded at runtime to get ETW working

Adding performance improvement

f35c01f

marcosd4h added 4 commits November 23, 2022 18:01

Adding support for older etw event versions

00fab51

Improving event enrichment phase

73af894

Removing test artifacts as this will added later

b8194dd

Adding event types to etw_process_events schema

12fdb90

Removing unnecesary warning pragmas

5ef2f0e

lcostantino reviewed Nov 24, 2022

View reviewed changes

Using wait with a predicate to avoid lost or spurious wakeups

099f97a

Adding krabsetw to libraries manifest

6695a8a

marcosd4h added 2 commits December 12, 2022 17:08

Adding krabsetw to libraries manifest

af70b2c

Adding krabsetw to libraries manifest

fa8498d

muffins reviewed Dec 15, 2022

View reviewed changes

marcosd4h added 7 commits December 23, 2022 17:21

Adding eventing tests for etw_process_events table

6b60238

Adding eventing tests for etw_process_events table

4c56676

Reducing test iterations to avoid CI test timeout

4662c3e

Adding integration test for etw_process_events table

9d04625

Adding popWait member to concurrent queue to fix issue on the shutdow…

d0b8f1f

…n logic. Improving shutdown logic on the ETW sessions runners. Adding more tests to cover the shutdown scenario. Adding integration test.

Fixing styling issue

4569635

Fixing CmakeLists file identation

a9b7183

zwass commented Jan 31, 2023

View reviewed changes

Addressing code review comments

3bc6d0d

zwass commented Feb 6, 2023

View reviewed changes

directionless approved these changes Feb 6, 2023

View reviewed changes

zwass merged commit 01d66ff into osquery:master Feb 7, 2023

	Column("type", TEXT, "Event Type"),
	Column("type", TEXT, "Event Type (ProcessStart, ProcessStop)"),


		namespace osquery {

		// Returns a reference to the single global EtwController instance

	// New events get store in the post-processing queue
	// New events get stored in the post-processing queue

Add ETW-based process events table for Windows #7821

Add ETW-based process events table for Windows #7821

Conversation

zwass commented Nov 17, 2022 • edited

zwass left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

defensivedepth Nov 24, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

directionless left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcosd4h Dec 12, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcosd4h Feb 3, 2023 • edited

Choose a reason for hiding this comment

mike-myers-tob commented Nov 22, 2022

lcostantino commented Nov 23, 2022

marcosd4h commented Nov 24, 2022

marcosd4h commented Nov 24, 2022

lcostantino commented Nov 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Smjert commented Dec 12, 2022 • edited

swannman commented Dec 12, 2022 • edited

Smjert commented Dec 12, 2022

marcosd4h commented Dec 12, 2022 • edited

Smjert commented Dec 12, 2022 • edited

muffins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcosd4h commented Dec 28, 2022

muffins commented Jan 7, 2023

marcosd4h commented Jan 9, 2023

zwass left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zwass left a comment

Choose a reason for hiding this comment

directionless left a comment

Choose a reason for hiding this comment

zwass commented Feb 6, 2023

directionless commented Feb 6, 2023

zwass commented Feb 6, 2023

directionless commented Feb 7, 2023

zwass commented Feb 7, 2023

zwass commented Nov 17, 2022 •

edited

defensivedepth Nov 24, 2022 •

edited

marcosd4h Dec 12, 2022 •

edited

marcosd4h Feb 3, 2023 •

edited

Smjert commented Dec 12, 2022 •

edited

swannman commented Dec 12, 2022 •

edited

marcosd4h commented Dec 12, 2022 •

edited

Smjert commented Dec 12, 2022 •

edited