[docs] mention timezone stuff

.clang-tidy

@@ -159,4 +159,6 @@ CheckOptions:
     value: '2'
   - key: 'readability-identifier-length.MinimumParameterNameLength'
     value: '2'
+  - key: 'cppcoreguidelines-avoid-do-while.IgnoreMacros'
+    value: 'true'
 ...

NEWS.md

@@ -98,6 +98,16 @@ Features:
   to a target timezone.
 * Added a `:convert-time-to` command that converts the
   timestamp of the focused log message to the given timezone.
+* Added the `:set-file-timezone` and `:clear-file-timezone`
+  commands to set the timezone for log messages that don't
+  include a zone in their timestamp.
+* Added the `options_path` and `options` columns to the
+  `lnav_file` table so you can see what options are applied
+  to a file.  Currently, the only option is the default
+  timezone that is set by the `:set-file-timezone` command.
+* Added the `config file-options` management command that
+  can be used to examine the options that will be applied
+  to a given file.
 
 Bug Fixes:
 * Binary data piped into stdin should now be treated the same

docs/source/cli.rst

@@ -127,6 +127,17 @@ Subcommands
    Print out the configuration options as JSON-Pointers and the
    file/line-number where the configuration is sourced from.
 
+.. option:: config file-options <path>
+
+   Print out the options that will be applied to the given file.  The
+   options are stored in the :file:`file-options.json` file in the
+   **lnav** configuration directory.  The only option available at
+   the moment is the timezone to be used for log message timestamps
+   that do not include a zone.  The timezone for a file can be set
+   using the :ref:`:set-file-timezone<set_file_timezone>` command
+   and cleared with the :ref:`:clear-file-timezone<clear_file_timezone>`
+   command.
+
 .. option:: format <format-name> get
 
    Print information about the given log format.
@@ -187,20 +198,20 @@ Environment Variables
 Examples
 --------
 
-  To load and follow the system syslog file:
+To load and follow the system syslog file:
 
-  .. prompt:: bash
+.. prompt:: bash
 
-    lnav
+   lnav
 
-  To load all of the files in :file:`/var/log`:
+To load all of the files in :file:`/var/log`:
 
-  .. prompt:: bash
+.. prompt:: bash
 
-    lnav /var/log
+   lnav /var/log
 
-  To watch the output of make:
+To watch the output of make:
 
-  .. prompt:: bash
+.. prompt:: bash
 
-    lnav -e 'make -j4'
+   lnav -e 'make -j4'

docs/source/config.rst

@@ -134,26 +134,26 @@ You can copy the code block, save it to a file in
 definition, see one of the definitions built into **lnav**, like
 `monocai <https://github.com/tstack/lnav/blob/master/src/themes/monocai.json>`_.
 
-  .. code-block:: json
+.. code-block:: json
-
+
-    {
+  {
-        "$schema": "https://lnav.org/schemas/config-v1.schema.json",
+      "$schema": "https://lnav.org/schemas/config-v1.schema.json",
-        "ui": {
+      "ui": {
-            "theme-defs": {
+          "theme-defs": {
-                "example1": {
+              "example1": {
-                    "vars": {
+                  "vars": {
-                        "black": "#2d2a2e"
+                      "black": "#2d2a2e"
-                    },
+                  },
-                    "styles": {
+                  "styles": {
-                        "text": {
+                      "text": {
-                            "color": "#f6f6f6",
+                          "color": "#f6f6f6",
-                            "background-color": "$black"
+                          "background-color": "$black"
-                        }
+                      }
-                    }
+                  }
-                }
+              }
-            }
+          }
-        }
+      }
-    }
+  }
 
 Reference
 ^^^^^^^^^

docs/source/cookbook.rst

@@ -1,4 +1,3 @@
-
 .. _Cookbook:
 
 Cookbook
@@ -69,9 +68,9 @@ Count client IPs in web access logs
 To count the occurrences of an IP in web access logs and order the results
 from highest to lowest:
 
-  .. code-block:: custsqlite
+.. code-block:: custsqlite
 
-    ;SELECT c_ip, count(*) as hits FROM access_log GROUP BY c_ip ORDER BY hits DESC
+   ;SELECT c_ip, count(*) as hits FROM access_log GROUP BY c_ip ORDER BY hits DESC
 
 
 Show only lines where a numeric field is in a range
@@ -81,9 +80,9 @@ The :ref:`:filter-expr<filter_expr>` command can be used to filter web access
 logs to only show lines where the number of bytes transferred to the client is
 between 10,000 and 40,000 bytes like so:
 
-  .. code-block:: custsqlite
+.. code-block:: custsqlite
 
-    :filter-expr :sc_bytes BETWEEN 10000 AND 40000
+   :filter-expr :sc_bytes BETWEEN 10000 AND 40000
 
 
 Generating a Report

docs/source/sqltab.rst

@@ -95,12 +95,22 @@ following columns are available in this table:
   :device: The device the file is stored on.
   :inode: The inode for the file on the device.
   :filepath: If this is a real file, it will be the absolute path.  Otherwise,
-    it is a symbolic name.  If it is a symbolic name, it can be UPDATEd so that
+    it is a symbolic name.  If it is a symbolic name, it can be UPDATEd
-    this file will be considered when saving and loading session information.
+    so that this file will be considered when saving and loading session
+    information.
+  :mimetype: The detected MIME type of the file.
+  :content_id: The hash of some unique content in the file.
   :format: The log file format for the file.
   :lines: The number of lines in the file.
   :time_offset: The millisecond offset for timestamps.  This column can be
     UPDATEd to change the offset of timestamps in the file.
+  :options_path: Options can be applied to files based on a path or glob
+    pattern.  If this file matches a set of options, the matching path/pattern
+    is available in this column and the actual options themselves are in the
+    :code:`options` column.
+  :options: The options that are applicable to this file.  Currently, the
+    only options available are for the timezone set by the
+    :ref:`:set-file-timezone<set_file_timezone>` command.
 
 lnav_file_metadata
 ------------------

docs/source/ui.rst

@@ -8,7 +8,7 @@ with status bars above and below, and the interactive prompt as the last line.
 
 .. figure:: lnav-ui.png
    :align: center
-   :alt: Screenshot of lnav showing a mix of syslog and web access_log messages.
+   :figwidth: 90%
 
    Screenshot of **lnav** viewing syslog and web access_log messages.
 
@@ -263,10 +263,10 @@ Markdown
 Files with an :code:`.md` (or :code:`.markdown`) extension will be treated as
 Markdown files and rendered separately.
 
-  .. figure:: lnav-markdown-example.png
+.. figure:: lnav-markdown-example.png
-     :align: center
+   :align: center
 
-     Viewing the **lnav** :file:`README.md` file.
+   Viewing the **lnav** :file:`README.md` file.
 
 
 DB

src/base/fs_util.cc

@@ -29,6 +29,8 @@
 
 #include "fs_util.hh"
 
+#include <stdlib.h>
+
 #include "config.h"
 #include "fmt/format.h"
 #include "itertools.hh"
@@ -37,6 +39,19 @@
 namespace lnav {
 namespace filesystem {
 
+Result<ghc::filesystem::path, std::string>
+realpath(const ghc::filesystem::path& path)
+{
+    char resolved[PATH_MAX];
+    auto rc = ::realpath(path.c_str(), resolved);
+
+    if (rc == nullptr) {
+        return Err(std::string(strerror(errno)));
+    }
+
+    return Ok(ghc::filesystem::path(resolved));
+}
+
 Result<auto_fd, std::string>
 create_file(const ghc::filesystem::path& path, int flags, mode_t mode)
 {

src/base/fs_util.hh

@@ -67,6 +67,9 @@ openp(const ghc::filesystem::path& path, int flags, mode_t mode)
     return open(path.c_str(), flags, mode);
 }
 
+Result<ghc::filesystem::path, std::string> realpath(
+    const ghc::filesystem::path& path);
+
 Result<auto_fd, std::string> create_file(const ghc::filesystem::path& path,
                                          int flags,
                                          mode_t mode);

src/file_collection.cc

@@ -820,4 +820,18 @@ file_collection::copy()
     retval.merge(*this);
     retval.fc_progress = this->fc_progress;
     return retval;
-}
+}
+
+size_t
+file_collection::other_file_format_count(file_format_t ff) const
+{
+    size_t retval = 0;
+
+    for (const auto& pair : this->fc_other_files) {
+        if (pair.second.ofd_format == ff) {
+            retval += 1;
+        }
+    }
+
+    return retval;
+}

src/file_collection.hh

@@ -203,6 +203,8 @@ struct file_collection {
         return this->fc_files.size() < get_limits().l_open_files;
     }
 
+    size_t other_file_format_count(file_format_t ff) const;
+
     file_collection rescan_files(bool required = false);
 
     void expand_filename(lnav::futures::future_queue<file_collection>& fq,

src/file_options.cc

@@ -65,7 +65,7 @@ static const typed_json_path_container<file_options_collection>
 bool
 file_options::operator==(const lnav::file_options& rhs) const
 {
-    return this->fo_default_zone == rhs.fo_default_zone;
+    return this->fo_default_zone.pp_value == rhs.fo_default_zone.pp_value;
 }
 
 json_string
@@ -117,6 +117,8 @@ file_options_collection::match(const std::string& path) const
 nonstd::optional<std::pair<std::string, file_options>>
 file_options_hier::match(const ghc::filesystem::path& path) const
 {
+    static const auto ROOT_PATH = ghc::filesystem::path("/");
+
     auto lookup_path = path.parent_path();
 
     while (true) {
@@ -127,7 +129,12 @@ file_options_hier::match(const ghc::filesystem::path& path) const
 
         auto next_lookup_path = lookup_path.parent_path();
         if (lookup_path == next_lookup_path) {
-            break;
+            if (lookup_path != ROOT_PATH) {
+                // remote paths won't end with root, so try that
+                next_lookup_path = ROOT_PATH;
+            } else {
+                break;
+            }
         }
         lookup_path = next_lookup_path;
     }

src/file_options.hh

@@ -43,9 +43,9 @@
 namespace lnav {
 
 struct file_options {
-    const date::time_zone* fo_default_zone{nullptr};
+    positioned_property<const date::time_zone*> fo_default_zone{nullptr};
 
-    bool empty() const { return this->fo_default_zone == nullptr; }
+    bool empty() const { return this->fo_default_zone.pp_value == nullptr; }
 
     json_string to_json_string() const;
 

src/internals/cmd-ref.rst

@@ -1216,14 +1216,14 @@
 
 .. _set_file_timezone:
 
-:set-file-timezone *zone* *pattern*
+:set-file-timezone *zone* *\[pattern\]*
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
   Set the timezone to use for log messages that do not include a timezone.  The timezone is applied to the focused file or the given glob pattern.
 
   **Parameters**
     * **zone\*** --- The timezone name
-    * **pattern\*** --- The glob pattern to match against files that should use this timezone
+    * **pattern** --- The glob pattern to match against files that should use this timezone
 
 
 ----

src/lnav.cc

@@ -1091,6 +1091,50 @@ struct refresh_status_bars {
     std::shared_ptr<top_status_source> rsb_top_source;
 };
 
+static void
+check_for_file_zones()
+{
+    auto with_tz_count = 0;
+    std::vector<std::string> without_tz_files;
+
+    for (const auto& lf : lnav_data.ld_active_files.fc_files) {
+        auto format = lf->get_format_ptr();
+        if (format == nullptr) {
+            continue;
+        }
+
+        if (format->lf_timestamp_flags & ETF_ZONE_SET
+            || format->lf_date_time.dts_default_zone != nullptr)
+        {
+            with_tz_count += 1;
+        } else {
+            without_tz_files.emplace_back(lf->get_unique_path());
+        }
+    }
+    if (with_tz_count > 0 && !without_tz_files.empty()) {
+        auto note
+            = attr_line_t("The file(s) without a zone: ")
+                  .join(
+                      without_tz_files, VC_ROLE.value(role_t::VCR_FILE), ", ");
+        auto um
+            = lnav::console::user_message::warning(
+                  "Some messages may not be sorted by time correctly")
+                  .with_reason(
+                      "There are one or more files whose messages do not have "
+                      "a timezone in their timestamps mixed in with files that "
+                      "do have timezones")
+                  .with_note(note)
+                  .with_help(
+                      attr_line_t("Use the ")
+                          .append(":set-file-timezone"_symbol)
+                          .append(
+                              " command to set the zone for messages in files "
+                              "that do not include a zone in the timestamp"));
+
+        lnav_data.ld_exec_context.ec_error_callback_stack.back()(um);
+    }
+}
+
 static void
 looper()
 {
@@ -1903,7 +1947,9 @@ looper()
                 if (rebuild_res.rir_completed
                     && (lnav_data.ld_log_source.text_line_count() > 0
                         || lnav_data.ld_text_source.text_line_count() > 0
-                        || !lnav_data.ld_active_files.fc_other_files.empty()))
+                        || lnav_data.ld_active_files.other_file_format_count(
+                               file_format_t::SQLITE_DB)
+                            > 0))
                 {
                     log_debug("initial build completed");
                     lnav_data.ld_initial_build = true;
@@ -1968,6 +2014,8 @@ looper()
                             lnav_data.ld_active_files.fc_files
                                 | lnav::itertools::for_each(
                                     &logfile::dump_stats);
+
+                            check_for_file_zones();
                         } else {
                             lnav_data.ld_files_view.set_selection(0_vl);
                         }
@@ -2229,7 +2277,11 @@ main(int argc, char* argv[])
         options_coll.foc_pattern_to_options[fmt::format(FMT_STRING("{}/*"),
                                                         var_path.in())]
             = lnav::file_options{
-                curr_tz,
+                {
+                    intern_string_t{},
+                    source_location{},
+                    curr_tz,
+                },
             };
         options_hier->foh_path_to_collection.emplace(ghc::filesystem::path("/"),
                                                      options_coll);

src/lnav.management_cli.cc

@@ -36,6 +36,7 @@
 #include "base/paths.hh"
 #include "base/result.h"
 #include "base/string_util.hh"
+#include "file_options.hh"
 #include "fmt/chrono.h"
 #include "fmt/format.h"
 #include "itertools.similar.hh"
@@ -80,6 +81,7 @@ struct subcmd_config_t {
 
     CLI::App* sc_config_app{nullptr};
     action_t sc_action;
+    std::string sc_path;
 
     static perform_result_t default_action(const subcmd_config_t& sc)
     {
@@ -117,6 +119,57 @@ struct subcmd_config_t {
         return {um};
     }
 
+    static perform_result_t file_options_action(const subcmd_config_t& sc)
+    {
+        auto& safe_options_hier
+            = injector::get<lnav::safe_file_options_hier&>();
+
+        if (sc.sc_path.empty()) {
+            auto um = lnav::console::user_message::error(
+                "Expecting a file path to check for options");
+
+            return {um};
+        }
+
+        safe::ReadAccess<lnav::safe_file_options_hier> options_hier(
+            safe_options_hier);
+
+        auto realpath_res = lnav::filesystem::realpath(sc.sc_path);
+        if (realpath_res.isErr()) {
+            auto um = lnav::console::user_message::error(
+                          attr_line_t("Unable to get full path for file: ")
+                              .append(lnav::roles::file(sc.sc_path)))
+                          .with_reason(realpath_res.unwrapErr());
+
+            return {um};
+        }
+        auto full_path = realpath_res.unwrap();
+        auto file_opts = options_hier->match(full_path);
+        if (file_opts) {
+            auto content = attr_line_t().append(
+                file_opts->second.to_json_string().to_string_fragment());
+            auto um = lnav::console::user_message::raw(content);
+            perform_result_t retval;
+
+            retval.emplace_back(um);
+
+            return retval;
+        }
+
+        auto um
+            = lnav::console::user_message::info(
+                  attr_line_t("no options found for file: ")
+                      .append(lnav::roles::file(full_path.string())))
+                  .with_help(
+                      attr_line_t("Use the ")
+                          .append(":set-file-timezone"_symbol)
+                          .append(
+                              " command to set the zone for messages in files "
+                              "that do not include a zone in the timestamp"));
+
+        return {um};
+    }
+
     subcmd_config_t& set_action(action_t act)
     {
         if (!this->sc_action) {
@@ -1094,6 +1147,15 @@ describe_cli(CLI::App& app, int argc, char* argv[])
             ->callback([&]() {
                 config_args.set_action(subcmd_config_t::blame_action);
             });
+
+        auto* sub_file_options = subcmd_config->add_subcommand(
+            "file-options", "print the options applied to specific files");
+
+        sub_file_options->add_option(
+            "path", config_args.sc_path, "the path to the file");
+        sub_file_options->callback([&]() {
+            config_args.set_action(subcmd_config_t::file_options_action);
+        });
     }
 
     {

src/lnav_commands.cc

@@ -382,7 +382,7 @@ com_set_file_timezone(exec_context& ec,
                   return elem.se_value;
               });
         try {
-            auto* tz = date::locate_zone(split_args[1]);
+            const auto* tz = date::locate_zone(split_args[1]);
             auto pattern = split_args.size() == 2
                 ? line_pair->first->get_filename()
                 : split_args[2];
@@ -401,7 +401,7 @@ com_set_file_timezone(exec_context& ec,
                          pattern.c_str(),
                          args[1].c_str());
                 coll.foc_pattern_to_options[pattern] = lnav::file_options{
-                    tz,
+                    {intern_string_t{}, source_location{}, tz},
                 };
 
                 auto opt_path = lnav::paths::dotlnav() / "file-options.json";
@@ -458,7 +458,8 @@ com_set_file_timezone_prompt(exec_context& ec, const std::string& cmdline)
                 auto match_res
                     = options_hier->match(line_pair->first->get_filename());
                 if (match_res) {
-                    file_zone = match_res->second.fo_default_zone->name();
+                    file_zone
+                        = match_res->second.fo_default_zone.pp_value->name();
                     pattern_arg = match_res->first;
                 }
 
@@ -512,7 +513,7 @@ com_clear_file_timezone(exec_context& ec,
             }
 
             log_info("clearing timezone for %s", args[1].c_str());
-            iter->second.fo_default_zone = nullptr;
+            iter->second.fo_default_zone.pp_value = nullptr;
             if (iter->second.empty()) {
                 coll.foc_pattern_to_options.erase(iter);
             }
@@ -5655,7 +5656,8 @@ readline_context::command_t STD_COMMANDS[] = {
             .with_parameter({"zone", "The timezone name"})
             .with_parameter(help_text{"pattern",
                                       "The glob pattern to match against "
-                                      "files that should use this timezone"}),
+                                      "files that should use this timezone"}
+                                .optional()),
         com_set_file_timezone_prompt,
     },
     {

src/log_format.cc

@@ -1066,7 +1066,7 @@ external_log_format::scan(logfile& lf,
 
         if (file_options) {
             this->lf_date_time.dts_default_zone
-                = file_options->second.fo_default_zone;
+                = file_options->second.fo_default_zone.pp_value;
         } else {
             this->lf_date_time.dts_default_zone = nullptr;
         }

src/log_format_impls.cc

@@ -112,7 +112,7 @@ class generic_log_format : public log_format {
 
             if (file_options) {
                 this->lf_date_time.dts_default_zone
-                    = file_options->second.fo_default_zone;
+                    = file_options->second.fo_default_zone.pp_value;
             } else {
                 this->lf_date_time.dts_default_zone = nullptr;
             }
@@ -551,7 +551,7 @@ public:
 
             if (file_options) {
                 this->lf_date_time.dts_default_zone
-                    = file_options->second.fo_default_zone;
+                    = file_options->second.fo_default_zone.pp_value;
             } else {
                 this->lf_date_time.dts_default_zone = nullptr;
             }
@@ -1227,7 +1227,7 @@ public:
 
             if (file_options) {
                 this->lf_date_time.dts_default_zone
-                    = file_options->second.fo_default_zone;
+                    = file_options->second.fo_default_zone.pp_value;
             } else {
                 this->lf_date_time.dts_default_zone = nullptr;
             }
@@ -1751,7 +1751,7 @@ public:
 
             if (file_options) {
                 this->lf_date_time.dts_default_zone
-                    = file_options->second.fo_default_zone;
+                    = file_options->second.fo_default_zone.pp_value;
             } else {
                 this->lf_date_time.dts_default_zone = nullptr;
             }

src/logfile.cc

@@ -201,8 +201,10 @@ logfile::file_options_have_changed()
         if (this->lf_file_options) {
             log_info(
                 "  tz=%s",
-                this->lf_file_options->second.fo_default_zone->name().c_str());
+                this->lf_file_options->second.fo_default_zone.pp_value->name()
-            if (this->lf_file_options->second.fo_default_zone != nullptr
+                    .c_str());
+            if (this->lf_file_options->second.fo_default_zone.pp_value
+                    != nullptr
                 && this->lf_format != nullptr
                 && !(this->lf_format->lf_timestamp_flags & ETF_ZONE_SET))
             {

src/yajlpp/yajlpp.hh

@@ -709,6 +709,11 @@ struct json_string {
         this->js_len = buf_pair.second;
     }
 
+    string_fragment to_string_fragment() const
+    {
+        return string_fragment::from_bytes(this->js_content, this->js_len);
+    }
+
     auto_mem<const unsigned char> js_content;
     size_t js_len{0};
 };

src/yajlpp/yajlpp_def.hh

@@ -1125,7 +1125,9 @@ struct json_path_handler : public json_path_handler_base {
 
     template<
         typename... Args,
-        std::enable_if_t<LastIs<const date::time_zone*, Args...>::value, bool>
+        std::enable_if_t<
+            LastIs<positioned_property<const date::time_zone*>, Args...>::value,
+            bool>
         = true>
     json_path_handler& for_field(Args... args)
     {
@@ -1138,7 +1140,11 @@ struct json_path_handler : public json_path_handler_base {
             try {
                 const auto* tz
                     = date::get_tzdb().locate_zone(value_str.to_string());
-                json_path_handler::get_field(obj, args...) = tz;
+                auto& field = json_path_handler::get_field(obj, args...);
+                field.pp_path = ypc->get_full_path();
+                field.pp_location.sl_source = ypc->ypc_source;
+                field.pp_location.sl_line_number = ypc->get_line_number();
+                field.pp_value = tz;
             } catch (const std::runtime_error& e) {
                 jph->report_tz_error(ypc, value_str.to_string(), e.what());
             }
@@ -1155,7 +1161,7 @@ struct json_path_handler : public json_path_handler_base {
                 const auto& field_def = json_path_handler::get_field(
                     ygc.ygc_default_stack.top(), args...);
 
-                if (field == field_def) {
+                if (field.pp_value == field_def.pp_value) {
                     return yajl_gen_status_ok;
                 }
             }
@@ -1166,7 +1172,7 @@ struct json_path_handler : public json_path_handler_base {
 
             yajlpp_generator gen(handle);
 
-            return gen(field->name());
+            return gen(field.pp_value->name());
         };
         return *this;
     }

test/expected/test_cmds.sh_b6a3bb78e9d60e5e1f5ce5b18e40d2f1662707ab.out

@@ -1455,7 +1455,7 @@ For support questions, email:
    
 
 
-:set-file-timezone zone pattern
+:set-file-timezone zone [pattern]
 ══════════════════════════════════════════════════════════════════════
   Set the timezone to use for log messages that do not include a
   timezone.  The timezone is applied to the focused file or the given

test/expected/test_logfile.sh_cc368d4b4bb6a9b9c79bd5a70ffa1f2d9d01e286.err

@@ -8,7 +8,7 @@
             Brazil/DeNoronha
             America/Barbados
             Asia/Baghdad
- = help: :set-file-timezone zone pattern
+ = help: :set-file-timezone zone [pattern]
          ══════════════════════════════════════════════════════════════════════
            Set the timezone to use for log messages that do not include a
            timezone.  The timezone is applied to the focused file or the given