Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#10959 closed enhancement (fixed)

PySlices patch

Reported by: mashbudn Owned by: robind
Priority: normal Milestone:
Component: wxPython Version: stable-latest
Keywords: Py, PyCrust, PySlices, slices, shell, PyShell, editor Cc: david.n.mashburn@…
Blocked By: Blocking:
Patch: yes

Description

I am finally uploading the changes I have added to PyCrust -- I took over as maintainer for the Py project in January.


The main additions to PyCrust include:

  • Better autoindentation
  • Some automatically loaded path features (cd, ls, pwd)
  • Support for magic command interpretation (magic.py, somewhat like what ipython does)
    • Support for the "func 1" -> "f(1)" comstruct
    • Multiple arguments written as: "func 1,2,3,4" -> "func(1,2,3,4)"
    • Ignores keywords and operators
    • Special support for cd, ls, pwd so they can be used like: "pwd" , "ls" , "ls *" , "ls *.py" , "cd .." , "cd /usr/"

In addition, I have also created a modified PyCrust called
PySlices which adds the following features:

  • A notebook-style interface ala SAGE or Mathematica
  • Use of colored markers instead of prompts (>>>)
    • Red markers = input (aka hot = editable)
    • Blue markers = output (aka frozen = not editable)
    • Black markers = grouping (to bundle respective input to its output)
    • Blocks of homogeneous lines are called "slices", so I refer to "input slices", "output slices", and "grouping slices"
  • Support for multi-line commands like:

"""a=1
b=2
c=3"""

  • Previous commands can be run IN PLACE!!
  • Restructuring of the keybindings:
    • RETURN adds a newline
    • ENTER and SHIFT+RETURN run commands
    • CTRL+RETURN calls manual auto-completion
  • Added keybindings for markers:
    • Ctrl-D divides an input "slice"
    • After selecting adjacent slices, Ctrl-M merges slices (if they are the same type)

I have found these changes to be both more productive and beginner-friendly. Most important are the simple abilities to and the ability to re-run previous commands in place. Specifically, I frequently use the ability to:

  • Run more than one command at once in a single slice
  • Re-run previous commands in place
  • Paste in any amount of code for debugging scripts
  • Run code from a whole script in a single "input slice"
  • Remove or add code in place
  • Break code into multiple slices using slice division feature, allowing the same code to be run in different sequences without any typing or editing
  • Perform tests on code without any saving or overwriting of code, unlike with a traditional editor
  • Have all executed code saved in the history
  • Create extra test code either in the same slice or in a new slice
  • Delete long outputs

Known bugs / Planned additions:

  • wx.STC (and underlying Scintilla) does not properly handle markers during Undo
    • I have been told that this functionality will not be added to Scintilla, as the current behavior is preffered in their project
    • We (I) will have to copy their undo-history hierarchy mechanism to make Undo function properly with markers so that slices will maintain their integrity
  • I am planning to implement a save format so that PySlices sessions can be saved and loaded (ala SAGE or MATHEMATICA)
  • I would like to implement a keybinding system with a graphical interface so that users can manually adjust the keybindings to suit their needs

Wish list / Far-reaching goals:

  • I would love to add more of ipython's features, but want to focus on the things that are most useful (unless someone REALLY wants to help in adding ipython's interpreter as a backend...)
  • I would love to be able to display graphics within the shell (especially for plotting with pylab)
  • I would really love to implement a good WYSIWYG equation editor in wxPython that can be used from within some sort of shell / notebook with 3 way conversion support between WYSIWYG math formatting, TeX, and sympy. The eventual goal would be to be able to use greek letters and mathematical notation as input and output in a python shell.
  • I have toyed with the idea of making a shell that can also interpret (with dynamic compilation) either Cython or C code for testing purposes

I know my wish list is pretty out there, but I just wanted to spell it out in case anyone wanted to help brainstorm ideas for implementing any of it!

My main focus will of course be in improving the stability and quality of the Py suite, with a focus on the new PySlices.

Personally, I now use PySlices on a daily basis and it performs the way I always wished that a shell would. I hope other people enjoy it too!

Attachments (10)

mypatch.patch download (142.0 KB) - added by mashbudn 5 years ago.
This is the patch that adds functionality to PyCrust and adds PySlices.
mypatchEdited.patch download (142.1 KB) - added by mashbudn 5 years ago.
PySlices.ico download (6.3 KB) - added by mashbudn 5 years ago.
PySlices_16.png download (733 bytes) - added by mashbudn 5 years ago.
PySlices_32.png download (1.8 KB) - added by mashbudn 5 years ago.
mypatch0976_final.patch download (184.9 KB) - added by mashbudn 5 years ago.
I think this patch is ready for commit. It is much improved and involved a lot of changes and cleanup.
pysuite0979.patch download (225.1 KB) - added by mashbudn 5 years ago.
Addition of File Save/Load and Code cleanup
pysuite0983.patch download (257.7 KB) - added by mashbudn 4 years ago.
Fixed a lot of bugs and quirk!
pysuite0983.2.patch download (257.7 KB) - added by mashbudn 4 years ago.
Fixed a lot of bugs and quirks!
pysuite0983_b.patch download (257.7 KB) - added by mashbudn 4 years ago.
Hopefully this patch will take without any conflicts…

Download all attachments as: .zip

Change History (22)

Changed 5 years ago by mashbudn

This is the patch that adds functionality to PyCrust and adds PySlices.

comment:1 follow-up: Changed 5 years ago by mashbudn


Oh, and I forgot to mention that I want to make both PyCrust and PySlices multithreaded ala wxipython and reinteract (which I just found today 7/6/09...)

The main function of this is so that the interpreter can be interrupted via the GUI.

comment:2 in reply to: ↑ 1 Changed 5 years ago by robind

Replying to mashbudn:


Oh, and I forgot to mention that I want to make both PyCrust and PySlices multithreaded ala wxipython and reinteract (which I just found today 7/6/09...)

The main function of this is so that the interpreter can be interrupted via the GUI.

It may not be necessary to use threads to do this, see this thread on wxPython-dev if you haven't seen it already. We're toying with some tweaks that will allow running a GUI from at least the console version of Python/IPython without blocking the interactive interpreter. The same may be doable with a wxGUI shell too The hook currently being used by the tweak is coming from readline so that may be console mode only, but I expect that a GUI shell could simulate that in some way.

comment:3 Changed 5 years ago by robind

Some issues and suggestions:

  • I've recently merged in the patch in #10874 and that causes a conflict with your patch. Now that I look closer at them it looks like they are doing basically the same thing. Could you please verify that? I don't care a whole lot which way is chosen so feel free to pick which ever you think is best when you update your patch.
  • Since PySlices.ico is a binary file it's not in the patch file. Please attach it here separately.
  • There is no myPyCrust module. (Used in PySlices.py)
  • You need to add a script file in the scripts dir for pyslices, update the Create* scripts there and also some files in distrib for installing it.
  • I don't like how the boolean useSlices is being passed around and used to control which kind of shell is used and other related stuff. What happens when there is a 3rd kind of shell in the future? It becomes even more messy. A better approach is to just pass in the class that should be used instead. Then the receiving function doesn't have to decide for itself which class to instantiate, it just uses the class that was given to it. For things besides the class that are currently dependent on useSlices (strings, icons, etc.) you can just make them be attributes or methods of the shell class instead, which further encapsulates the details so the frame classes and etc. don't need to know or care about which they are using.
  • I don't think that it should cd to my home dir automatically at startup. If I'm already in a folder where I'm working on things and start a pycrust from there, I want it to stay there. It's easy enough to do "cd ~" myself in the shell once I get there if that is where I want to be.
  • Don't use wxversion in this code. You need to assume that if the proper wx.py package was found on the Python path then all the rest of that same version of wx will be found too.
  • If a slice contains an import statement and other lines in the same slice use the imported module, it is not able to show the autocompeltes and calltips for the items in that module. I realize that this is because the module hasn't actually been imported yet, but it makes it feel like it is broken. Perhaps you could watch for import statements and sorta "pre-import" them so the introspection helpers will work within the same slice as expected.
  • Shift-Return to execute the slice feels very awkward. Maybe swap that and Ctrl-Return? Maybe use an F-key? Or maybe something like hitting Return twice in a row would make sense. Another idea that might help people transition is to have two modes that could be toggled between. In Slice mode the Return key adds a line to the current slice and Ctrl-Return executes the slice. In Shell mode it is the opposite: Return executes the slice and Ctrl-Return adds a line. That way in Shell mode it is more like PyShell where each statement is executed as it is entered unless you explicitly add more lines to the slice. I think that will be much more intuitive until people get used to the slice concept.
  • Speaking of which, it might be a good idea to display a couple paragraphs describing the slice concept the first time it is run by the user. (And be able to redisplay it from an item on the help menu.) It could just be text written to the shell window although a tip dialog would work too.
  • I think that having the group and the slices and the code folds all be collapsible makes the margin way too messy and confusing, and I really miss the Python >>> and ... prompts. So I suggest that you make the code-folding margin be able to be turned on or off from a menu item (and its state remembered when the settings are saved/loaded.) For the slice margin I suggest that you change it from the collapsible markers to STC_MARK_ARROWS and STC_MARK_DOTDOTDOT for the input slices so it looks like the standard Python prompts, and then something else for the output slices.
  • It seems to take a little longer to popup the autocomplete for large namespaces, (the wx module for example) did you change anything there? An idea for a future enhancement would be to cache the autocomp strings if there are more than some N number of items.
  • It would be nice to have more visual separation between the input and output slices so it is easier to see at a glance which is which. Perhaps give the output slices a STC_MARK_BACKGROUND marker (so the whole line is shaded, not just where there is text.)
  • Watch your line lengths. I'm not a "it must be 80 characters or less, no exceptions" kind of programmer, but if I often have to scroll horizontally or widen my editor window to read a line when it is already showing 90-100 characters I get frustrated. A few long lines here and there where it just isn't easily dividable or wrapped into multiple lines is ok, but that should be the exception, not the rule.
  • Since you are adding the special commands like ls and cd, it would be nice to go ahead and allow arbitrary shell commands to be run directly from the pyshell. For example, using a ! prefix could indicate to run the rest of the line through the command shell and display the output, like "!ls -al" The next step after that is to allow the definition of aliases where the value of the alias could be any shell or pyshell line, something like "alias ll = !ls -al"

comment:4 Changed 5 years ago by cprecord

Just as another comment when reading the notes about this patch.

I don't think that the part for "CTRL+RETURN calls manual auto-completion" should be applied, because the current Ctrl+Space combo is a rather standard key combination for this sort of feature amongst other various applications. Changing that functionality would also not really provide any gains and would only lead to confusion for all those used to using Ctrl+Space.

Changed 5 years ago by mashbudn

Changed 5 years ago by mashbudn

Changed 5 years ago by mashbudn

Changed 5 years ago by mashbudn

comment:5 Changed 5 years ago by mashbudn

Wow! Thanks for all the suggestions!

First of all, I am reading through all the comments, but I first wanted to send the PySlices.ico file and let you know that the "wxversion" and "myPyCrust" were local changes that I tried to remove before submitting the patch. I apparently missed them the 12 times I scanned through everything! Anyway, I removed them from this new patch.

I also changed the boolean "useSlices" boolean into a string, called "shellName". I tried to use a class, but there was an import order issue in frame.py. I think a name string should solve the more-than-two-shell-options problem.

I checked on the autocompletion feature that Cody and I each wrote, and I thought I would just ask Cody:


Cody:

Do you think it is valuable to test for the keywords of (if|else|elif|for|while|def|class) ?


The version I wrote just looks for a colon as the last non-whitespace character and avoids using re. If you like your version better, though, I am happy to keep it instead.


I'll be looking over and thinking about the other changes and suggestions, I just wanted to get back to you quickly with the easy fixes :)

BTW, do you have a suggestion for running a local copy of this stuff without resorting to a special import command for the "py" module (hence my use of "from myPyCrust import py" instead of "from wx import py")? Any suggestions would be greatly appreciated and would save me from maintaining a version for the svn trunk and a version for myself...

Thanks,
-David


P.S.
I also wanted to let you know that after we iron out the issues here, I have fixed the undo problem I mentioned before.

comment:6 Changed 5 years ago by mashbudn

One more note about the autoindent issue:

My version will respect non-4-space indentation...

so, if you've typed:

"for i in range(10):\n print i"

and you hit return, you get two spaces on the next line as well. It will also respect tabs.

Is this feature useful?

comment:7 Changed 5 years ago by mashbudn

Robin,

I'm submitting a highly improved patch. I think this one is ready to go now. I did all the cleanup work that you suggested.

Here are my responses to your comments. I put my replies in "--" :

  • I've recently merged in the patch in #10874 and that causes a conflict with your patch. Now that I look closer at them it looks like they are doing basically the same thing. Could you please verify that? I don't care a whole lot which way is chosen so feel free to pick which ever you think is best when you update your patch.

--I took some ideas from Cody's patch and came up with a new version that is more complete than either was before.

  • You need to add a script file in the scripts dir for pyslices, update the Create* scripts there and also some files in distrib for installing it.

--Done, I think.

  • I don't like how the boolean useSlices is being passed around and used to control which kind of shell is used and other related stuff. What happens when there is a 3rd kind of shell in the future? It becomes even more messy. A better approach is to just pass in the class that should be used instead. Then the receiving function doesn't have to decide for itself which class to instantiate, it just uses the class that was given to it. For things besides the class that are currently dependent on useSlices (strings, icons, etc.) you can just make them be attributes or methods of the shell class instead, which further encapsulates the details so the frame classes and etc. don't need to know or care about which they are using.

--I changed the bool to a string, which I think is probably the cleanest way to do it.

  • I don't think that it should cd to my home dir automatically at startup. If I'm already in a folder where I'm working on things and start a pycrust from there, I want it to stay there. It's easy enough to do "cd ~" myself in the shell once I get there if that is where I want to be.

--Let's work on this after this patch goes through... this is the same behavior that the old PyCrust had

  • If a slice contains an import statement and other lines in the same slice use the imported module, it is not able to show the autocompeltes and calltips for the items in that module. I realize that this is because the module hasn't actually been imported yet, but it makes it feel like it is broken. Perhaps you could watch for import statements and sorta "pre-import" them so the introspection helpers will work within the same slice as expected.

--Again, maybe we can look into this later. For now, I'd just suggest running the import statements in their own slice so autocomplete will work.
--When you think of PySlices as a shell/editor hybrid, I think it makes sense

  • Shift-Return to execute the slice feels very awkward. Maybe swap that and Ctrl-Return? Maybe use an F-key? Or maybe something like hitting Return twice in a row would make sense.

--Now, both Ctrl-Return, Shift-Return, and Numpad-Enter all execute in the normal mode

  • Another idea that might help people transition is to have two modes that could be toggled between. In Slice mode the Return key adds a line to the current slice and Ctrl-Return executes the slice. In Shell mode it is the opposite: Return executes the slice and Ctrl-Return adds a line. That way in Shell mode it is more like PyShell where each statement is executed as it is entered unless you explicitly add more lines to the slice. I think that will be much more intuitive until people get used to the slice concept.

--Made shell mode... shell mode has normal python prompts as the markers and two Returns executes the slice. I think the two-returns is simple enough.
--Ctrl-Return and Shift-Return now just do newlines

  • Speaking of which, it might be a good idea to display a couple paragraphs describing the slice concept the first time it is run by the user. (And be able to redisplay it from an item on the help menu.) It could just be text written to the shell window although a tip dialog would work too.

--I made a tutorial that shows up on startup unless you disable it (and instructions for this are in the tutorial)

  • I think that having the group and the slices and the code folds all be collapsible makes the margin way too messy and confusing. So I suggest that you make the code-folding margin be able to be turned on or off from a menu item (and its state remembered when the settings are saved/loaded.)

--I made the code-folding margin diabled by default and enable-able from the View Menu

  • I really miss the Python >>> and ... prompts. For the slice margin I suggest that you change it from the collapsible markers to STC_MARK_ARROWS and STC_MARK_DOTDOTDOT for the input slices so it looks like the standard Python prompts, and then something else for the output slices.

--I added this to "Shell Mode". I stuck with folding margins for output because everything else just didn't look good. For "Slices Mode" I stayed with the folding markers, because I'm going for a Mathematica/Sage kind of feel.
--I think the "Shell Mode" markers should be what you're looking for

  • It seems to take a little longer to popup the autocomplete for large namespaces, (the wx module for example) did you change anything there?

--Not sure why... It seems to take the same amount of time for me...

  • An idea for a future enhancement would be to cache the autocomp strings if there are more than some N number of items.

--Good idea! We'll work on that later.

  • It would be nice to have more visual separation between the input and output slices so it is easier to see at a glance which is which. Perhaps give the output slices a STC_MARK_BACKGROUND marker (so the whole line is shaded, not just where there is text.)

-- I added a subtle coloring to output. I didn't add it to the tutorial, though, because I thought it made it hard to read.

  • Watch your line lengths. I'm not a "it must be 80 characters or less, no exceptions" kind of programmer, but if I often have to scroll horizontally or widen my editor window to read a line when it is already showing 90-100 characters I get frustrated. A few long lines here and there where it just isn't easily dividable or wrapped into multiple lines is ok, but that should be the exception, not the rule.

--Done

  • Since you are adding the special commands like ls and cd, it would be nice to go ahead and allow arbitrary shell commands to be run directly from the pyshell. For example, using a ! prefix could indicate to run the rest of the line through the command shell and display the output, like "!ls -al" The next step after that is to allow the definition of aliases where the value of the alias could be any shell or pyshell line, something like "alias ll = !ls -al"

--Great idea! I'll work on that another time.

  • I don't think that the part for "CTRL+RETURN calls manual auto-completion" should be applied, because the current Ctrl+Space combo is a rather standard key combination for this sort of feature amongst other various applications. Changing that functionality would also not really provide any gains and would only lead to confusion for all those used to using Ctrl+Space.

--I changed it to Ctrl-Shift-H.
--I would kind of like to combine the Autocomplete / Calltip / History complete into one operation, but for now, you can just use Ctrl-Space, Ctrl-Shift-Space, and Ctrl-Shift-H repectively

Changed 5 years ago by mashbudn

I think this patch is ready for commit. It is much improved and involved a lot of changes and cleanup.

comment:8 Changed 5 years ago by mashbudn

Ah! I was wrong about the cd issue... I had a startup script that did the same thing that was confusing me. I fixed the patch to remove the cd to home.

Hope this patch is easier to read now!

Changed 5 years ago by mashbudn

Addition of File Save/Load and Code cleanup

comment:9 Changed 5 years ago by mashbudn

  • Keywords slices added

I have made some very significant changes since the last patch I submitted.

The most significant change is that PySlices now supports saving and loading of a special "*.pyslices" text format. This format is also valid python.

The second significant change is that I did a major code cleanup, separating crust.py into crust.py and crustslices.py. These serve PyCrust and PySlices respectively.

I also added a script for a standalone shell form of PySlices called "PySlicesShell.py" (very creative, I know...)

There were also a few bug fixes.

All changes are listed in detail in CHANGES.txt

The new patch reflects the current version number 0.9.7.9

Thanks,
-David

Changed 4 years ago by mashbudn

Fixed a lot of bugs and quirk!

Changed 4 years ago by mashbudn

Fixed a lot of bugs and quirks!

comment:10 Changed 4 years ago by robind

  • Owner changed from David Mashburn to robind
  • Status changed from new to accepted

Will review patch soon.

Changed 4 years ago by mashbudn

Hopefully this patch will take without any conflicts...

comment:11 Changed 4 years ago by RD

  • Resolution set to fixed
  • Status changed from accepted to closed

(In [63479]) Apply PySlices patch from David Mashburn. Closes #10959

comment:12 Changed 4 years ago by RD

(In [63480]) Apply PySlices patch from David Mashburn. Closes #10959

Note: See TracTickets for help on using tickets.