feat(previews): enhance scientific file previews and LaTeX compilation diagnostics

- Expanded the scientific file preview capabilities to include new formats such as MATLAB, Matrix Market, and FITS images, with improved visualization options for arrays and tables.
- Updated the LaTeX compilation process to retain detailed diagnostics logs, separating final errors from resolved warnings, and ensuring that compilation failures are clearly reported.
- Enhanced the handling of user selections in scientific previews, allowing for more precise dataset and slice management.
- Improved the LaTeX editor's user interface to support new view modes and better manage compilation states, including error handling and user notifications.
- Updated documentation to reflect changes in preview capabilities and LaTeX compilation processes, ensuring users are informed of new features and best practices.

Author: timothykassis

docs/file-previews.md

@@ -23,6 +23,24 @@ context without changing the document. Proposed replacements still use the
 existing diff review. A suggestion is not a verified compilation; recompile
 after accepting it. Missing-context responses are billed like other AI calls.
 
+The LaTeX editor offers **Source**, **Split**, and **PDF** views and remembers
+the selected view and split width. PDF previews start fitted to the pane and
+resize with it; manual zoom and **Fit width** remain available. **Compile on
+save** runs after an explicit save (it does not autosave as you type).
+
+The PDF is marked out of date after source edits, and a failed build keeps the
+previously displayed preview with a failure notice. Compiler errors in the open
+file are clickable in the log; stale line locations cannot launch AI fixes.
+The diagnostic gutter uses the final compiler pass, so warnings resolved by
+reruns disappear. Bibliography failures still fail the build even if a later
+TeX pass produces a PDF. Long logs retain both their beginning and final summary.
+
+Select source and choose **Edit with AI** (⌘K / Ctrl+K). For chapter files with a
+`% !TEX root = ../main.tex` comment, AI assistance also reads the root preamble.
+Finish the diff review before saving, compiling, or requesting another AI edit.
+Edits typed during a save stay unsaved, and the editor's **Close** button prompts
+before discarding them.
+
 ### Genomics & sequences
 | Format | Extensions | Viewer |
 |---|---|---|
@@ -47,10 +65,46 @@ after accepting it. Missing-context responses are billed like other AI calls.
 | Format | Extensions | Viewer |
 |---|---|---|
 | Single-cell (AnnData) | `h5ad` `h5ad.gz` | Structured card — obs/var columns, layers, embeddings you can color by any column |
-| HDF5 | `h5` `hdf5` | Group / dataset tree with shapes, dtypes, attributes |
-| Parquet | `parquet` | Schema + first rows |
-| NumPy | `npy` `npz` | Shape, dtype, min/max/mean, value preview (each array in an `.npz`) |
-| NetCDF | `nc` `nc4` `cdf` | Dimensions, variables, and global attributes |
+| HDF5 | `h5` `hdf5` | Group / dataset tree plus selectable numeric heatmaps and line plots |
+| Parquet | `parquet` | Schema, searchable/sortable row sample, selectable numeric scatter plot |
+| NumPy | `npy` `npz` | Array selector, heatmap / line plot, multidimensional slice controls, displayed-value statistics |
+| NetCDF | `nc` `nc4` `cdf` | Dimensions, root variables, attributes and numeric slice plots; masked values remain gaps |
+| MATLAB | `mat` | Dense numeric variables from v4–v7.2 files; v7.3 uses the HDF5 dataset viewer and storage-axis order |
+| Matrix Market | `mtx` | Dense or sparse matrix heatmap; sparse sources are sampled before conversion to a small dense grid |
+| FITS images | `fits` `fit` `fts` | Image HDU selector, slice controls, heatmaps / line plots with FITS value scaling |
+
+### Data tables & workbooks
+| Format | Extensions | Viewer |
+|---|---|---|
+| Excel | `xlsx` | Sheet selector, table search / sort and numeric scatter plots; first row supplies headers, formulas show saved values |
+| Arrow IPC / Feather v2 | `arrow` `ipc` `feather` | File or stream schema, row sample and numeric scatter plots |
+| JSON Lines | `jsonl` `ndjson` | Sampled records as columns; nested values shown as JSON, with numeric scatter plots |
+| SQLite | `sqlite` `sqlite3` `db` | Read-only ordinary-table selector, row sample, search / sort and numeric scatter plots |
+
+Tables show up to **200 rows × 50 columns**; search, sorting and plots use only
+that sample. Large cell text is capped at 512 characters. Where a format lacks
+a cheap exact row count, the viewer explicitly reports that the total was not
+scanned. SQLite views and virtual tables are excluded. JSON Lines columns are
+inferred from displayed records. Charts omit missing/nonnumeric pairs; empty
+cells and booleans are not treated as zero.
+
+Numeric arrays show a line plot for vectors and a heatmap for matrices. For
+higher dimensions, the slice control walks the leading dimensions in C order
+while displaying the last two axes. Indices start at zero. Plots use at most
+**1,024 vector points** or a **128 × 128 strided grid**. The viewer labels
+sampling, missing values, and statistics from the displayed slice/sample;
+these are not full-dataset statistics. Hover over a heatmap cell or plot point
+to inspect its value. Striding can miss isolated features in large arrays.
+
+HDF5/NetCDF and NumPy `.npy` previews read slices; Parquet reads a limited row
+batch. XLSX expanded content, Arrow files/batches, individual NPZ arrays and
+classic MATLAB files/variables have a 64 MiB preview ceiling. Matrix Market is
+limited to 64 MiB and two million stored entries / axis. JSON Lines scans at
+most 8 MiB, with a 1 MiB per-line limit. File/container metadata and codec work
+can still require additional memory. Oversized inputs show an explanation and
+remain available for download or analysis. Array/variable and sheet/table lists
+are bounded and show truncation notices. NumPy object arrays are never unpickled;
+HDF5 external links and external/virtual dataset storage are not followed.
 
 ### Bio-imaging
 | Format | Extensions | Viewer |
@@ -70,4 +124,4 @@ after accepting it. Missing-context responses are billed like other AI calls.
 
 ## Unsupported previews
 
-There are no dedicated viewers for whole-slide imaging (`.svs`, `.ndpi`), GenBank/NEXUS annotated records, multi-channel OME-TIFF compositing or cross-file DICOM series stacking. Unrecognized files use the text fallback where possible; download binary formats or process them with the agent.
+There are no dedicated viewers for whole-slide imaging (`.svs`, `.ndpi`), GenBank/NEXUS annotated records, multi-channel OME-TIFF compositing or cross-file DICOM series stacking. The new data viewers do not cover legacy Excel `.xls`, ODS, Feather v1, FITS table HDUs, MATLAB cells/structs or nested NetCDF groups. NPZ previews accept NumPy header versions 1 and 2. Unrecognized files use the text fallback where possible; download binary formats or process them with the agent.

server/src/api/sandbox.ts

@@ -733,19 +733,27 @@ export async function registerSandboxRoutes(app: FastifyInstance): Promise<void>
   );
 
   // --- generic scientific-file previews (chem/structure/...) via Python helper ---
-  app.get<{ Querystring: { path: string; kind: string } }>("/sandbox/sci-summary", async (req, reply) => {
+  app.get<{ Querystring: { path: string; kind: string; key?: string; slice?: string } }>("/sandbox/sci-summary", async (req, reply) => {
     try {
       if (!sciHelperFor(req.query.kind)) {
         reply.code(400);
         return { detail: `Unknown kind: ${req.query.kind}` };
       }
+      const { key, slice } = req.query;
+      if ((key !== undefined && (typeof key !== "string" || key.length > 1024 || key.includes("\0"))) ||
+          (slice !== undefined && (!/^\d+$/.test(slice) || !Number.isSafeInteger(Number(slice)))) ||
+          ((key !== undefined || slice !== undefined) && !["arrays", "tables"].includes(req.query.kind))) {
+        reply.code(400);
+        return { detail: "Invalid scientific preview selection" };
+      }
       const target = safePath(req.query.path);
       if (!fs.existsSync(target) || !fs.statSync(target).isFile()) {
         reply.code(404);
         return { detail: "File not found" };
       }
       const res = await requestPreview(reply, {
         projectId: currentProjectId(), target, script: sciHelperFor(req.query.kind)!.script, command: "summarize",
+        params: key !== undefined || slice !== undefined ? [key ?? "", slice ?? "0"] : undefined,
       });
       reply.header("Cache-Control", "private, no-cache");
       if (res.timedOut) {

server/src/api/sci-helpers.ts

@@ -16,14 +16,15 @@ import path from "node:path";
 import { execFile } from "node:child_process";
 import { HELPERS_DIR, helperPython } from "../helpers-env.ts";
 
-export type SciKind = "chem" | "structure" | "massspec" | "arrays" | "imaging";
+export type SciKind = "chem" | "structure" | "massspec" | "arrays" | "imaging" | "tables";
 
 const KIND_TO_SCRIPT: Record<string, string> = {
   chem: "chem_helper.py",
   structure: "structure_helper.py",
   massspec: "massspec_helper.py",
   arrays: "arrays_helper.py",
   imaging: "imaging_helper.py",
+  tables: "tables_helper.py",
 };
 
 /** Wall-clock ceiling for a single preview helper invocation. */
@@ -79,6 +80,7 @@ export interface HelperResult {
 
 /** Absolute helper script path for a known kind, or null if the kind is unrecognized. */
 export function sciHelperFor(kind: string): { script: string } | null {
+  if (!Object.hasOwn(KIND_TO_SCRIPT, kind)) return null;
   const file = KIND_TO_SCRIPT[kind];
   if (!file) return null;
   return { script: path.join(HELPERS_DIR, file) };

server/src/api/sci-previews.ts

@@ -39,6 +39,9 @@ async function version(request: PreviewRequest): Promise<string> {
   return JSON.stringify(await Promise.all([
     fileVersion(request.target),
     fileVersion(request.script),
+    // A live SQLite database may have newer committed data in its WAL.
+    ...(/\.(db|sqlite3?)$/i.test(request.target)
+      ? [fileVersion(`${request.target}-wal`).catch(() => "missing-wal")] : []),
     ...[helperPython(), path.join(HELPERS_DIR, "pyproject.toml"), path.join(HELPERS_DIR, ".venv", "pyvenv.cfg"), path.join(HELPERS_DIR, "uv.lock")]
       .map((file) => fileVersion(file).catch(() => "missing")),
   ]));
@@ -52,7 +55,7 @@ export async function getPreview(request: PreviewRequest, signal?: AbortSignal):
     try {
       const output = dir ? path.join(dir, "image") : "";
       const params = request.params ?? [];
-      const args = request.command === "summarize" ? ["summarize", request.target]
+      const args = request.command === "summarize" ? ["summarize", request.target, ...params]
         : request.command === "render" ? ["render", request.target, params[0] ?? "0", output, params[1] ?? "-"]
         : ["embedding", request.target, params[0], params[1] ?? "-", request.cacheDir!, output];
       const result = await runHelperScript(request.script, args, undefined, workSignal);