-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for graphics in the terminal #4763
base: master
Are you sure you want to change the base?
Conversation
d5a1c08
to
f11ce16
Compare
This sounds like it might have a significant performance impact. Have you actually tested the performance of this PR?
That sounds extremely hacky, which I am not a fan of. This will likely just cause an endless heap of issues with things like selection, so it's not something we can just throw in and forget about.
This seems to just pile on a heap of code to support a bunch of image protocols. I don't like the idea of adding thousands of lines to Alacritty for something with so little use. If we support any protocol it should be the simplest one without any performance impact. I see no benefit in supporting multiple formats.
What about support for devices below OpenGL 3.3? This is something we would like to look into for the future so adding more code that requires at least OpenGL 3.3+ does not seem ideal.
I see little reason for adding memoffset/lazy_static usually, but I haven't looked at the code yet. |
Same |
I launched vtebench three times, in the current Running $ base64 ./target/release/alacritty > /tmp/data
$ wc -l /tmp/data
635257 /tmp/data
# graphics branch
$ perf stat -e cycles,instructions,branches,branch-misses ./target/release/alacritty -e cat /tmp/data
Performance counter stats for './target/release/alacritty -e cat /tmp/data':
4,590,935,968 cycles
10,125,002,121 instructions # 2.21 insn per cycle
2,038,270,623 branches
4,651,069 branch-misses # 0.23% of all branches
0.854657772 seconds time elapsed
0.774462000 seconds user
0.466697000 seconds sys
# master branch
$ perf stat -e cycles,instructions,branches,branch-misses ./target/release/alacritty -e cat /tmp/data
Performance counter stats for './target/release/alacritty -e cat /tmp/data':
4,692,460,970 cycles
10,053,624,431 instructions # 2.14 insn per cycle
2,035,973,894 branches
4,795,961 branch-misses # 0.24% of all branches
0.855561156 seconds time elapsed
0.762745000 seconds user
0.493408000 seconds sys
I changed the NBSP character with a
Without comments and tests, the support for the iTerm2 protocol is less than 100 lines of code.
The only performance impact of this protocol is an extra branch in the
The fallback uses |
This issue is now fixed. |
That doesn't justify adding it. If Alacritty supports any graphics protocol, there should be one and it should be the simplest one available. |
I'm also getting the following error:
|
What is the output of |
96 or something like that iirc. |
I uploaded a fix in 387a224 to check if this fixes the problem. If this works, we may need to generate the code of the fragment shader dynamically, in order to use the 32 units if they are available in the hardware. |
@ayosec did you test the iTerm2 protocol with ranger? previewed images are kept on screen and not cleared, so new images are overlayed on top of the old ones, and images are kept in the background when text is rendered in the preview pane. Another issue is that the iterm2 graphics is working with tmux as long as the image file is small, but with large files, it looks like I'm hitting this, which is not an Alacritty issue but Alaciritty is spamming with warning messages like this:
You should be able replicate with:
Note that this issue doesn't happen when running ranger directly without going through a tmux session, and not when running in an ssh session. |
Ranger relies on overwriting images with spaces. The discussion in #910 rejected my first approach to be able to have this functionality. However, if we can change the implementation in this patch to use As a work-around, images can be replaced with the █ character (reversed with
This error happens because the base64 stream is incomplete. Did you use iTerm2 imgcat? It seems that tmux requires specific sequences to send images. Anyways, since iTerm2 will not be accepted, maybe it's not worth it to spend time debugging the issue. |
I can replicate the issue with the linked imgcat script and an image. I won't attach the image here because I'm not sure about its license but I can send it via email.
That's unfortunate, it would have been nice to have it. |
I guess it's better to wait for sixel to support 24-bit colors? |
Sixel isn't likely to change. |
I have updated the patch with the following changes:
|
Can your implementation of sixel convert 24bit colors into 256 colors and display a video efficiently? |
18fd66e
to
0a41635
Compare
@nixpulvis There are a number of use cases I take advantage of regularly:
That said, I think the current situation works well enough provided @ayosec doesn't mind maintaining a soft fork. I couldn't expect y'all to pull in code you weren't interested in maintaining. I also appreciate alacritty's simplicity compared to a lot of other terminals, and I assume the same attitude that resulted in your opinions on this PR is why it doesn't have a lot of other stuff I probably wouldn't want included. Big thanks to all the maintainers of Alacritty, and another big thank you ayosec for maintaining this fork :) |
The reason that people demand images in terminal is that major GUI toolkits like Gtk and Qt are worse than terminal user interface in many ways. Gtk and Qt break backward compatibility every few years in the name of innovations which tend to lead to more complexity instead of more simplicity. I evaluated Gtk and Qt and other GUI toolkits and decided to use a TUI library because I didn't want to migrate my application to new versions of Gtk or Qt every few years. And, Gtk and Qt are difficult to use in good programming languages. Gtk and Qt are useable on Python and C++, all of which are bad languages. I found a TUI library that has a saner programming model than either Gtk or Qt, and doesn't break backward compatibility every few years in the name of innovations. If we have a good GUI library that easily allows creation of GUI applications better than terminal UI applications, then this issue wouldn't exist. |
This is not an argument to add image support to terminals. If anything it's an argument why it should not be added.
So to avoid bloating UI toolkits, whose purpose is rendering textures, you want to instead increase the complexity of terminal emulators, which never had any business rendering images?
You can open as many images as you want with sixel and then just go through them. So image count alone isn't really an argument for/against simple image viewers.
This is just using a terminal multiplexer as a replacement for a proper window manager.
GUI as root is just as bad as terminal as root. If anything a file manager temporarily elevating privileges to root has less resource access than a terminal emulator.
E-Mails, just like Websites, unfortunately aren't text-based protocols anymore. There is no "good" solution here, HTML E-Mail is just always atrocious. |
With sixel/timg I can list out thumbnails of a large directory of images all at once in a single view. I'm not aware of a GUI image viewer that defaults to a full page thumbnail view, and going through separate windows or a single window with one at a time would take way too long, so before I had sixel integrated into my setup I'd end up opening a file manager each time. It isn't the end of the world, but it's nice not having to do that 20+ times a day.
I do use terminal apps pretty heavily for a lot of my computing, and I use tmux as a window manager for them, but I'm not sure there's a proper window manager solution to having three panes with vim on the left and one on the top right with mutt sized to 30% of the height, and then quickly getting an image to fill the spot on the bottom right and only have it there when the terminal is in focus.
My file manager doesn't offer the ability to switch to root or other users. There's also a quote by a Gnome dev at the top of https://wiki.archlinux.org/title/Running_GUI_applications_as_root that I agree with, but even ignoring that end of things, I still wouldn't be able to quickly access images locked to a separate user without having to become root and then fish them out. I actually remembered a related use case that I hadn't included before: viewing images on other computers over ssh.
I've managed everything but my work email pretty well exclusively by terminal for many years. Until I incorporated sixel support I had to save the html attachments and then open them with a web browser, but haven't had to do that once since. I don't see the email exactly like the sender intends (if such a thing is possible), but I'm not missing any of the content either. I fully agree that HTML email is garbage. Anyway, I wasn't trying to argue, apologies if it came off like I was. I realize that everyone uses software their own way, and yours should take priority being that you're developing it. I thought it was worth sharing that there are reasonable use cases for some sort of terminal image solution as having sixel has improved my workflow a ton, but I'm sure others out there would have their workflow improved by gui tabs and I'm happy I don't need to unconfigure those in each Alacritty install. I also wanted an excuse to thank you all for the amazing project! |
Yeah I'm not doubting that it might have helped you. I just don't personally ever see any arguments that make me feel like they're of sufficient value to compromise on one of the core principles of terminal emulation. |
I don't think I really understand what that principle is. If Alacritty is an emulation, it is presumably an emulation of a historical terminal. Why then shouldn’t it include every feature that the historical terminal had? DEC terminals had both Sixel and ReGIS graphics protocols (for bitmap and vector graphics, respectively). Tektronics terminals had their own graphics protocol which was emulated by terminals from Wyse and others. The IBM 2250 had better support for vector graphics than for text; you either had to pay extra to get the optional character generator or you would draw characters using vector graphics. |
I think any tiling WM should do the trick fairly easily. From the top of my head i3wm for X11-based, sway for Wayland-based, or some tiling-capable WM like awesome should be able to do this too without having to enforce tiling on all windows
Terminal were HID peripherals that evolved until being ultimately replaced by the screen+keyboard+mouse we have nowadays. The choice of emulating based on standard hardware is for backward compatibility and integrating with/benefiting from an ecosystem that has been there for decades. That being said, you can't emulate any and all terminal fonction because you'd end up getting closer and closer to just having a software screen or a graphics library. We already have those and they use more efficient approaches than a text-based protocol. This leaves us with having to draw a line between what is considered "basic text-based interface to interact with users and display things" and what is better left to graphics applications. That line is the responsibility of maintainers and each project has its own. |
If each project has it’s own threshold, then it isn’t “one of the core principles of terminal emulation.” |
I'll admit my previous response is a bit all over the place... The core principle is being text-oriented as opposed to being/providing a graphical framework. You won't see a purely graphical terminal emulator without text support but you can find lots of terminal emulator without graphics support (never say never I know but I'd like to see a real life use case of such emulator) |
Here is a picture of someone using a terminal in 1970: Or how about this one, from 1971: (source) Of course, Alacritty doesn’t emulate the IBM 2250. I point out these particular photos because on this terminal, text really was optional. You had to pay tens of thousands of dollars extra to get a character generator! Alacritty mostly emulates the DEC terminals. But the DEC terminals had both vector and bitmap graphics support, and plenty of applications used them. There were CAD programs and mapping (GIS) programs. There were engineers and scientists studying graphs. There were games. Surely an emulator should emulate those capabilities accurately, even if they are inefficient by comparison with modern graphics apis. And even if it is hard to find photographs of people actually using them. Does anyone know what type of terminals were shown in Farewell Etaoin Shrdlu? Not many images, but definitely some early WYSIWYG text rendering in 1978. |
I don’t know anything about how they implemented it though. |
I've been comparing TE image strategies for a while now, so I'll add my $0.02 on sixels. Pros:
Cons:
I consider sixels to be the Latin of image protocols. From a technical viewpoint it's not great, but like many TE things, it's old and solidified a long time ago. Plus it's more or less neutral ground - no one developer has the upper hand. |
@JasonGantner That assumes I'm managing those terminal sessions as separate windows and not in tmux. I use sway on one of my boxes and it's great for making non-terminal windows behave a bit more like they're in a tmux session, but it doesn't replace a lot of why I use tmux. For a window manager to solve the scenario I put forth it would need to be aware of tmux, the space each pane takes up, and be able to place a window above the terminal window but below others when those others are above the terminal one. |
My biggest issue, not to derail this thread too much, is that sometimes I want tiled windows and sometimes I want floating windows, and to this day I still haven't made sway work with me to achieve my desires. There should be an easy way to tell sway before hand that I want a floating window when launched, but sadly I don't know if it's possible. This is what I mean by window managers still seem lacking, since if they were truly good at their jobs, launching an application to view things like email, images, and pdfs wouldn't seem as cumbersome and staying in the terminal for things it really wasn't designed for would seem less desirable. |
I think GUI toolkits should become stable and simple. However, for the foreseeable future, terminal file managers are superior to GUI file managers for advanced linux users. It's good to have any serviceable image protocol on a terminal emulator for image preview on terminal file manager. We already have other terminal emulators that support at least one terminal image protocol. So, I can just use another terminal emulator if I want image preview. According to my research, it's okay to have a simple terminal image protocol that fills character cells instead of pixels. If terminal UI toolkis have to worry only about character cells, terminal image can be simple. |
Terminal emulator is a user interface. Thus, we should approach this from the perspective of UI desinger. Character-cell based GUI is simple and effective. I only wish it had proper images that fill character cells. Then, it would be a good UI. Perhaps, for this to happen properly, we need a new UI platform... I don't know. |
@amano-kenji If you haven't already and you're able to, try building https://github.com/ayosec/alacritty/tree/graphics - you get alacritty + sixel, and in my experience it works better than the few alternatives I've compared it to (specifically blackbox terminal emulator and konsole). |
we can package this fork if we want, I suppose, provided ayosec is willing to mantain it. |
That's a fair summary of sixel, but I'd add one more positive and one more negative.
|
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
Jesus, could we please not have this whole discussion here? It's like: What if unstoppable force meets immovable object... I would love to see graphics support in Alacirtty, because it is useful for me to viewing graphics on the server without downloading them, but every project has certain rules and assumptions and Alacritty is not a Swiss army knife. I asked myself what was needed for any protocol related to graphics support to be included in the project or what was wrong with the current solutions, because I was curious both about chrisduerr's opinion and because I was interested in reading about technical matters. He replied that no protocol meets the necessary assumptions and is simply a no-no. And although it was not satisfactory for me and I was counting on some complex technicalities, despite this, I understand his position. He and kchibisov take care of Alacritty and there is no third maintainer who would take an equal share from them and also take care of the graphics-related code, and every functionality we have meets their standards and does not fall any lower. I don't care about ligatures, and I have Sway and Zellij for tabs and windows. I've tried using other terminals and have them installed as backups, but there's something about the color reproduction or font drawing that I can't explain, but it bothers me in them and not in Alacritty, that's why I stay here and I really like simplicity. ayosec is a real hero, because without him and his fork I wouldn't be able to use Alacritty and still be able to check images on the server or html e-mails from time to time, when I need to. Every update he make, I compile it and check if everything works... and so far I haven't noticed anything wrong (to my eye). And I will continue to use his work as long as he keeps doing it (I don't know how to program and I don't see anyone sending any code to the ayosec branch to correct or add anything). Every time I visit here, I'm still afraid that at some point he'll stop updating, because he'll be tired of all this shit. And if making a sacrifice to Cthulhu would accomplish anything, I would do it. Could we limit the discussions here to maintaining this branch and possibly technical discussions on how to improve anything, without trying to involve chrisduerr and kchibisov? There is probably nothing more they need to add on their part, and further attempts here look like a war of attrition. |
From your comment, @roland-5, I'm curious if the current maintainers would accept the author of this PR as another maintainer to "take care of the graphics-related code" within Alacritty, if the author of this PR has time and interest in doing so… A huge thank you to the current maintainers for Alacritty and the author of this PR to keep supporting the graphics-feature branch! |
That just doesn't make any sense. |
Thanks for thinking about it and taking the time to reply, Chris. Too bad it doesn't make any sense. |
This patch adds support for graphics in the terminal. Graphics can be sent as Sixel images.
New features
Sixel parser
The Sixel parser is based on the SIXEL GRAPHICS EXTENSION chapter in a DEC manual.
The support is complete except for the pixel aspect ratio parameters. According to the manual, a Sixel image can specify that it expects a specific shape for the pixels in the device, but in none of terminals that I checked these parameters had any effect: they always assume a 1:1 ratio. Also, I didn't find any Sixel generator that emits a different ratio. To avoid extra complexity in the parser, it always assume 1:1 when the image is built.
There are two new terminal modes:
SixelScrolling
(80
)If enabled, new graphics are inserted at the cursor position, and they can scroll the grid.
If disabled, new graphics are inserted at top-left, and they are limited by the height of the window.
SixelPrivateColorRegisters
(1070
)If enabled, every Sixel parser has its own color palette.
If disabled, Sixel images can share a color palette.
Initially I didn't plan to support this mode, since it seems to be specific to xterm, but when I was testing applications using Sixel I found that mpv uses it to reuse a palette between video frames. Since mpv is based on libsixel, I guess that this feature could be used by more applications.
Both modes are enabled by default.
The function to convert HLS colors to RGB is a direct port of the implementation of the same function in xterm. I verified that the function emits the same results in all combinations of values 0, 30, 60, 90, and 100 in every color component. Only a few of these combinations were left in the tests to reduce the noise in the code.
To test the parser there are Sixel images generated with 3 different applications. For each one, there is a
.rgba
file in the same directory with the expected RGBA values. The commands to produce these files are inalacritty_terminal/tests/sixel/README.md
.Byte
90
as DCSSixel images using byte
90
as DCS are not supported. DCS can be eitherESC P
or90
, but thevte
crate only recognizesESC P
. I guess that this is because90
can be a continuation byte in a UTF-8 sequence (two most significant bits are10
), so it can be a valid input from users.Xterm has the same limitation. I don't expect that many applications depends on it.
ppmtosixel
is an exception. It uses90
to start the Sixel data, and it has to be replaced if we want to see an image generated by it.It is still interesting to test
ppmtosixel
because it was written in 1991, long before Sixel was added to most (if not all) terminal emulators.