System and method for interactive sewing operations

Abstract

The invention relates to a system (10, 20, 30) for interactive sewing operations. The system comprises a sewing machine (10) to be operated by a user for effecting a sewing operation on at least one workpiece by using at least one thread; said sewing machine having a plurality of operating elements interacting with the at least one workpiece and the at least one thread. The system further comprises at least one sensor (20, 21, 22) for generating machine feedback data representative of an operational state of at least one of a) the plurality of operating elements, b) the at least one thread and c) the at least one workpiece, and for transmitting said machine feedback data. The system further comprises an augmented reality device (30) adapted for receiving said machine feedback data from the at least one sensor (20, 21, 22) and presenting corresponding user feedback data to at least one of the user's senses. The augmented reality device (30) is adapted for being placed between the sewing machine (10) and the user.

Description

[0001] "See-through AR device" (P273)

[0002] System and method for interactive sewing operations

[0003] The invention relates to a system and method for interactive sewing operations.

[0004] Sewing machines are used for different kinds of sewing operations.

[0005] The terms "sewing" or "sewing machine" are used for any kind of stitch formation in a flat and typically flexible workpiece formed from material such as leather, polymer foils, metal foils, woven or non-woven fabrics or any other kind of textile material using one or more threads. Typically, a series of stitches are formed in one, two or more layers of such flat and typically flexible material in order to

[0006] - attach small decorative workpieces (appliques) to a larger base workpiece,

[0007] - sew two workpieces together by forming a seam by many consecutive stitches along overlapping edge regions of the two workpieces,

[0008] - sew a stitch pattern on a base workpiece (embroidery),

[0009] - sew a stack of three or more layers of fabric together (quilting),

[0010] etc.

[0011] It is an object of the invention to provide a user of a system adapted to such sewing operations (sewing, embroidery, quilting, etc.) with automated support during the operations, typically by providing feedback to the user in the form of information and guidance for handling the workpiece(s) and for sewing machine configuration, i.e. machine parameter settings.

[0012] According to the invention, a system (10, 20, 30) for interactive sewing operations is provided, which system comprises

[0013] a sewing machine (10) to be operated by a user for effecting a sewing operation on at least one workpiece by using at least one thread;

[0014] said sewing machine having a plurality of operating elements interacting with the at least one workpiece and the at least one thread;

[0015] at least one sensor (20, 21, 22) for generating machine feedback data representative of an operational state of at least one ofa) the plurality of operating elements,

[0016] b) the at least one thread,

[0017] c) the at least one workpiece; and

[0018] for transmitting said machine feedback data;

[0019] an augmented reality device (30) adapted for receiving said machine feedback data from the at least one sensor (20, 21, 22) and presenting corresponding user feedback data to at least one of the user's senses;

[0020] wherein said augmented reality device (30) is adapted for being placed between the sewing machine (10) and the user.

[0021] The machine feedback data and user feedback data may comprise at least one of the following information displayed in the augmented reality device as added virtual information in addition to real information:

[0022] - preview of the seam in real time

[0023] - feedback information relating to seam quality (good, bad)

[0024] - needle frequency (too high, too low)

[0025] - workpiece (fabric) velocity relative to stitch plate (too high, too low)

[0026] - emphasizing obstacles in the workpiece (fabric), such as puckering or buttonholes

[0027] - emphasizing the approaching end of a seam being sown

[0028] - emphasizing the approaching end of the workpiece / fabric.

[0029] Moreover, the machine feedback data and user feedback data may comprise at least one of the following information displayed in the augmented reality device alone or as added virtual information in addition to real information:

[0030] Preview of a needle entry point:

[0031] This information shows to the user the exact point on the fabric where the needle will enter. This is similar to a laser pointer, but more flexible in its application.

[0032] Preferably, the preview of the needle entry point can be switched on and off.

[0033] Preferably, at least one of color, brightness, size and shape of the preview of the needle entry point can be changed.

[0034] Preferably, the size and / or shape of the dot or line shown to the user can be customized. For instance, the preview of the needle entry point can be in the form of crosshairs.

[0035] Preferably, the at least one sensor (20, 21, 22) constitutes a fabric height detector forproducing fabric height information which can be taken into account for presenting the preview of the needle entry point.

[0036] Preferably, the at least one sensor (20, 21, 22) constitutes a fabric color and / or brightness detector for producing fabric color and / or brightness information which can be taken into account for presenting a color and / or brightness of the preview of the needle entry point. Preview of a sewing stitch:

[0037] This information shows to the user the currently selected sewing stitch type on the fabric currently being used.

[0038] Preferably, the preview of the sewing stitch is updated in real time as the sewing machine is being operated. As a result, the sewing stitch preview moves with the fabric.

[0039] Preferably, at least one of color, brightness, size and shape of the preview of the sewing stitch can be changed.

[0040] Preferably, the line thickness of the sewing stitch type shown to the user can be changed. Preferably, the at least one sensor (20, 21, 22) constitutes a fabric height detector for producing fabric height information which can be taken into account for presenting the preview of the sewing stitch. As a result, the sewing stitch preview can take the height and / or the 3D shape of the fabric (such as ripples) into account for presenting the preview of the sewing stitch.

[0041] Preferably, the at least one sensor (20, 21, 22) constitutes a fabric color and / or brightness and / or fabric type detector for producing fabric color and / or brightness and / or fabric type information which can be taken into account for presenting a color and / or brightness of the preview of the sewing stitch.

[0042] Displaying guidance lines and / or guidance grids:

[0043] This information shows to the user a configurable variety of guidance lines and / or grids on the fabric currently being used.

[0044] Preferably, the displayed guidance lines and / or guidance grids are updated in real time as the sewing machine is being operated. As a result, the displayed guidance lines and / or guidance grids move with the fabric.

[0045] Preferably, at least one of color, brightness, size and shape of the displayed guidance lines and / or guidance grids can be changed.

[0046] Preferably, at least one of line thickness, number of lines, grid spacing, angles of lines withrespect to fabric features and / or with respect to other lines of the guidance lines and / or guidance grids shown to the user can be changed.

[0047] Preferably, the at least one sensor (20, 21, 22) constitutes a fabric height detector for producing fabric height information which can be taken into account for displaying the guidance lines and / or guidance grids. As a result, the displayed guidance lines and / or guidance grids can take the height and / or the 3D shape of the fabric (such as ripples) into account for presenting the guidance lines and / or guidance grids.

[0048] Preferably, the at least one sensor (20, 21, 22) constitutes a fabric color and / or brightness and / or fabric type detector for producing fabric color and / or brightness and / or fabric type information which can be taken into account for selecting a color and / or brightness of the displayed guidance lines and / or guidance grids.

[0049] Preview of an embroidery pattern or quilting pattern:

[0050] This information shows to the user the currently selected embroidery or quilting pattern on the fabric currently being used.

[0051] Preferably, the preview of the embroidery or quilting pattern includes at least one of colors and shapes of the embroidery or quilting pattern to be produced by a given sequence of stitches.

[0052] Preferably, the at least one sensor (20, 21, 22) constitutes an embroidery frame / hoop pose (position and orientation) detector producing embroidery frame (hoop) pose information which can be taken into account for displaying the selected embroidery pattern on the fabric fixed in the embroidery frame. As a result, the displayed embroidery pattern can take the pose of the embroidery frame into account for presenting the selected embroidery pattern on the fabric.

[0053] Preferably, the at least one sensor (20, 21, 22) constitutes feature and / or marker detector identifying and localizing feature and / or marker information on a fabric. Preferably, the features are inherent to the fabric while the markers are added / attached to the fabric. As a result, a selected embroidery or quilting pattern can be previewed on a piece of fabric prior to being hooped (fixed in the frame / hoop) or fixed on a quilting frame.

[0054] Preferably, the preview of the embroidery or quilting pattern is updated in real time as the sewing machine is being operated in embroidery or quilting mode. As a result, the already stitched out or quilted part of the embroidery or quilting pattern can be omitted from the preview.Preferably, the at least one sensor (20, 21, 22) constitutes a fabric color and / or brightness and / or fabric type detector for producing fabric color and / or brightness and / or fabric type information.

[0055] Preferably, the brightness of the embroidery pattern preview can be configured or automatically adapted based on fabric color and / or fabric brightness and / or fabric type. Preferably, the at least one sensor (20, 21, 22) constitutes an illumination intensity detector for producing illumination intensity information.

[0056] Preferably, the brightness of the embroidery pattern preview can be configured or automatically adapted based on fabric illumination intensity.

[0057] Preferably, the at least one sensor (20, 21, 22) constitutes a hoop position detector for producing information about the pose (position and orientation) of an embroidery hoop relative to the sewing machine. The hoop position detector may be a camera.

[0058] Alternatively or additionally, such information about the pose (position and orientation) of the hoop relative to the sewing machine can be obtained by data exchange communication between the sewing machine and an embroidery module attached to the sewing machine and holding the embroidery hoop in a defined pose (position and orientation).

[0059] As a result, the displayed embroidery pattern can take the pose of the embroidery hoop relative to the sewing machine into account for presenting the selected embroidery pattern on the fabric.

[0060] Displaying ruler or distance measurement tools:

[0061] This information shows to the user virtual measurement tools such as rulers or stencils. Preferably, the ruler is displayed on the fabric while sewing or in the hoop while embroidering.

[0062] Alternatively or additionally, the ruler may be displayed on a housing part of the machine such as the machine bed and / or next to the machine on any flat surface.

[0063] Preferably, the at least one sensor (20, 21, 22) constitutes a finger tip position detector for producing information about the position of at least one finger tip. As a result, an adaptable ruler may be displayed between two finger tips. By moving the finger tips away from each other and towards each other, the ruler may be extended and contracted, respectively.

[0064] Displaying warnings:

[0065] This information shows to the user warnings with respect to a plurality of operating elements

[0066] 5of the sewing machine.

[0067] The warning may be displayed on the fabric while sewing or in the hoop while embroidering. Alternatively or additionally, the warning may be displayed, i.e. virtually projected, on a housing part of the machine such as the machine bed and / or next to the machine on any flat surface. The display or virtual projection of an operating element specific warning on a real surface may be selected according to a location of the operating element within the sewing machine and / or with a symbol identifying the operating element.

[0068] For instance, an arrow pointing toward the operating element position or a closed line around the operating element position may be used.

[0069] Displaying instructions (step by step) about machine use:

[0070] This information shows to the user instructions about using the sewing machine.

[0071] Preferably, this information can be displayed as simple graphics or videos.

[0072] Preferably, the at least one sensor (20, 21, 22) constitutes a detector for detecting one step within a sequence of sewing steps. As a result, a sewing step specific instruction for a subsequent sewing step can be displayed when the user has completed a previous sewing step.

[0073] Preview of a sewing pattern:

[0074] This information shows to the user a selected or planned sewing pattern on the fabric currently being used.

[0075] As a result, the user can cut along the pattern lines or transfer the lines from the virtual display / projection to the fabric, preferably by using real (physical) markers.

[0076] Preview of a buttonhole (buttonhole assistant):

[0077] This information shows to the user the location and shape of a buttonhole on the fabric being used.

[0078] Preferably, the at least one sensor (20, 21, 22) constitutes a detector for detecting one step within a sequence of buttonhole sewing steps. The sensor may be a camera. As a result, a buttonhole sewing step specific instruction for a subsequent buttonhole sewing step can be displayed when the user has completed a previous buttonhole step.

[0079] The preview may be a static or dynamic preview of a buttonhole.

[0080] Preferably, the at least one sensor (20, 21, 22) constitutes a detector for detecting the degree of completion of one step within a sequence of buttonhole sewing steps. The sensor may bea camera. As a result, a camera in the sewing machine or the augmented reality device can be used to provide feedback to the user on the status of the buttonhole sewing progress and detect an endpoint of a buttonhole pattern.

[0081] Preferably, a camera is used to correct the position and direction of stitches during buttonhole sewing.

[0082] Preferably, the augmented reality device displays feedback during buttonhole sewing such as arrows, frames, circles or brief text messages.

[0083] Displaying virtual user interface (Ul) elements:

[0084] This information displays user interface (Ul) elements that do not exist in reality.

[0085] Preferably, such virtual Ul elements comprise at least one of buttons, knobs, sliders, labels, warnings and information boxes.

[0086] Preferably, such virtual Ul elements are displayed at a location at least on the sewing machine, on the fabric being used and in mid-air.

[0087] Preferably, the at least one sensor (20, 21, 22) constitutes a finger tip position detector for producing information about the position of at least one finger tip. As a result, a virtual Ul element ruler may be operated by at least one finger tip. By moving two finger tips parallel to each other and in opposite directions, a virtual knob may be rotated.

[0088] Displaying augmented user interface (Ul) elements:

[0089] This information provides existing Ul elements with customized or bespoke information depending on and relevant to a currently required use of the sewing machine.

[0090] Preferably, such customized / bespoke information is a virtual pattern on a virtual Ul element or a real Ul element.

[0091] Preferably, such virtual Ul elements or real Ul elements comprise at least one of buttons, knobs, sliders, labels, warnings and information boxes.

[0092] Providing a virtual magnifying glass:

[0093] This information provides a virtual zoom in a selectable part of the field of view.

[0094] Preferably, the sewing machine has several onboard cameras providing several different views of the sewing machine.

[0095] This allows virtual magnification of a selectable part of a field of view of any such camera.

[0096] Providing virtual transparency:

[0097] This information provides virtual transparency of at least a part of the sewing machine.Preferably, the sewing machine has several onboard cameras providing several different views of the sewing machine.

[0098] This allows a selectable part of the sewing machine to be made virtually transparent.

[0099] Preferably, an onboard camera is provided at the sewing machine.

[0100] Preferably, the onboard camera takes images of the fabric from a different angle than the augmented reality device.

[0101] Preferably, these images taken by the onboard camera are used to make parts of the sewing machine at least partially transparent, for instance totally transparent or semi-transparent.

[0102] Provide assistance for multi hooping:

[0103] This information provides assistance for multi-hooping when large patterns are to be embroidered.

[0104] Preferably, a previous partial embroidery pattern is detected.

[0105] Preferably, within a hoop, a position and rotation of a subsequent section is calculated. Preferably, a preview of the subsequent calculated section is displayed.

[0106] This helps making sure the fabric is hooped correctly and there is enough space for the next section. As a result, multi-hooping becomes possible by localizing the previously stitched section with sufficient precision and adding the rest of the embroidery pattern perfectly. Detecting and identifying accessories:

[0107] This information enables detection of installed or attached sewing machine accessories such as presser foot or needle.

[0108] Preferably, this information is used to display information and recommendations for better suited accessories or warnings of possible danger originating from detected accessory in combination with current sewing machine settings.

[0109] Preferably, the augmented reality device (30) comprises a see-through device adapted for viewing reality information and comprising a means for displaying additional information in addition to said reality information for providing augmented reality information.

[0110] Preferably, the augmented reality device (30) comprises means for attaching it to the user's body at a location close to at least one of the user's sense organs.

[0111] Preferably, the at least one sensor (21, 22) is located at the sewing machine (10) and / or at the augmented reality device (30).Preferably, the augmented reality device (30) comprises a pair of eyeglasses which can be attached to the user's head.

[0112] Preferably, the augmented reality device (30) comprises optical display means in at least one of its two transparent eyeglass portions, said optical display means providing an optical display of said user feedback data as a superposition to the user's field of view through the eyeglass portions.

[0113] The augmented reality device may comprise a head-up display, a transparent thin film display, a contact lens display, a holographic display or a virtual retinal display.

[0114] Preferably, the augmented reality device (30) comprises a pair of earphones or speakers which can be attached to the user's head.

[0115] Preferably, the at least one sensor is a device sensor (22) provided at said pair of eyeglasses. The device sensor (22) may comprise a field of view tracking sensor (camera) for providing field of view data or an eye tracking sensor for providing eye orientation data or a depth sensor for providing distance information.

[0116] Preferably, the pair of eyeglasses comprises a data processing unit for processing data generated by said at least one device sensor.

[0117] In particular, field of view data and eye orientation data are processed in order to determine the user's viewing direction within the field of view when the user wears the pair of eyeglasses.

[0118] Preferably, the at least one sensor is a machine sensor (21) provided at said sewing machine (10).

[0119] Preferably, the sewing machine (10) and the augmented reality device (30) are connected by a data communication path (50).

[0120] Preferably, the sewing machine (10) and the augmented reality device (30) each comprise a transmitter-receiver means (51, 52) for establishing a wireless data transmission path (50) for data communication between the sewing machine (10) and the augmented reality device (30).

[0121] Preferably, the augmented reality device (30) comprises a rechargeable battery as a power supply.Preferably, the sewing machine (10) comprises means for receiving said augmented reality device (30) in a storing position when it is not being used.

[0122] Preferably, the sewing machine (10) and the augmented reality device (30) comprise complementary contact means contacting each other and providing an electric charging connection when said augmented reality device (30) is in said storing position.

[0123] The charging connection may provide electrical contact between a power supply of the sewing machine 10 and a rechargeable battery of the augmented reality device (30).

[0124] The electrical contact may be provided by a galvanic connection for charging or may be provided by a contact-less connection for charging via inductive coupling or capacitative coupling (both short range) or via resonant inductive coupling (mid-range).

[0125] According to the invention, a method for interactive sewing operations is provided, which method comprises the following steps:

[0126] providing a system for interactive sewing operations comprising a sewing machine (10) and an augmented reality device (30) as previously defined;

[0127] operating the sewing machine (10) by a user for effecting a sewing operation on at least one workpiece by using at least one thread and a plurality of operating elements of the sewing machine (10) interacting with the at least one workpiece and the at least one thread;

[0128] while using the augmented reality device (30) adapted for receiving machine feedback data from the at least one sensor (20, 21, 22) and for presenting corresponding user feedback data to at least one of the user's senses, said augmented reality device (30) being placed between the sewing machine (10) and the user;

[0129] generating, by the at least one sensor (20, 21, 22), machine feedback data representative of an operational state of at least one of

[0130] a) the plurality of operating elements,

[0131] b) the at least one thread,

[0132] c) the at least one workpiece;

[0133] transmitting said machine feedback data from the at least one sensor (20, 21, 22) to the augmented reality device (30); andpresenting, by the augmented reality device (30), corresponding user feedback data to at least one of the user's senses.

[0134] Preferably, the augmented reality device (30) comprises a see-through device adapted for viewing reality information and comprising a means for displaying additional information in addition to said reality information for providing augmented reality information; wherein using said augmented reality device (30) by the user comprises viewing said sewing machine (10) through said augmented reality device (30) together with said corresponding user feedback data.Fig. 1 schematically shows a system 10, 20, 30 for interactive sewing operations.

[0135] The system comprises a sewing machine 10 to be operated by a user for effecting a sewing operation on at least one workpiece by using at least one thread. The sewing machine 10 has a plurality of operating elements (not shown) interacting with the at least one workpiece (not shown) and the at least one thread (not shown).

[0136] The system comprises at least one sensor 20, 21, 22 for generating machine feedback data representative of an operational state of at least one of a) the plurality of operating elements, b) the at least one thread and c) the at least one workpiece, and for transmitting said machine feedback data.

[0137] The system comprises an augmented reality device 30 adapted for receiving said machine feedback data from the at least one sensor 20, 21, 22 and for presenting corresponding user feedback data to at least one of the user's senses such as eyes and / or ears. The augmented reality device 30 is adapted for being placed between the sewing machine 10 and the user, preferably in front of the user's head 40.

[0138] Sensor 20 is a stand-alone sensor having its own periphery 19. Sensor 21 is a machine sensor where the sensor periphery is provided by the sewing machine 10. Sensor 22 is a device sensor where the sensor periphery is provided by the augmented reality device 30.

[0139] The sewing machine 10 and the augmented reality device 30 are connected by a data communication path 50. The sewing machine 10 and the augmented reality device 30 each comprise a transmitter-receiver means 51, 52 for establishing a wireless data transmission path 50 for data communication between the sewing machine 10 and the augmented reality device 30.List of reference signs

[0140] 10 sewing machine

[0141] 19 periphery of sensor 20

[0142] 20 sensor (stand-alone sensor)

[0143] 21 machine sensor (at sewing machine)

[0144] 22 device sensor (at augmented reality device) 30 augmented reality device (AR device)

[0145] 40 user's head

[0146] 50 data communication path (wireless)

[0147] 51 first transmitter-receiver means

[0148] 52 second transmitter-receiver means

Claims

"See-through AR device" (P273)Claims1. A system (10, 20, 30) for interactive sewing operations, comprising:a sewing machine (10) to be operated by a user for effecting a sewing operation on at least one workpiece by using at least one thread;said sewing machine having a plurality of operating elements interacting with the at least one workpiece and the at least one thread;at least one sensor (20, 21, 22) for generating machine feedback data representative of an operational state of at least one ofa) the plurality of operating elements,b) the at least one thread,c) the at least one workpiece; andfor transmitting said machine feedback data;an augmented reality device (30) adapted for receiving said machine feedback data from the at least one sensor (20, 21, 22) and presenting corresponding user feedback data to at least one of the user's senses;wherein said augmented reality device (30) is adapted for being placed between the sewing machine (10) and the user.

2. The system of claim 1, wherein said augmented reality device (30) comprises a see-through device adapted for viewing reality information and comprising a means for displaying additional information in addition to said reality information for providing augmented reality information.

3. The system of claim 1 or 2, wherein said augmented reality device (30) comprises means for attaching it to the user's body at a location close to at least one of the user's sense organs.

4. The system of any one of claims 1 to 3, wherein said at least one sensor (21, 22) is located at the sewing machine (10) and / or at the augmented reality device (30).

5. The system of any one of claims 1 to 4, wherein said augmented reality device (30) comprises a pair of eyeglasses which can be attached to the user's head.

6. The system of any one of claims 1 to 5, wherein said augmented reality device (30) comprises optical display means in at least one of its two transparent eyeglass portions, said optical display means providing an optical display of said user feedback data as a superposition to the user's field of view through the eyeglass portions.

7. The system of any one of claims 1 to 6, wherein said augmented reality device (30) comprises a pair of earphones or speakers which can be attached to the user's head.

8. The system of any one of claims 1 to 7, wherein said at least one sensor is a device sensor (22) provided at said pair of eyeglasses.

9. The system of claim 8, wherein said pair of eyeglasses comprises a data processing unit for processing data generated by said at least one device sensor.

10. The system of any one of claims 1 to 9, wherein said at least one sensor (20) is a machine sensor (21) provided at said sewing machine (10).

11. The system of any one of claims 1 to 10, wherein said sewing machine (10) and said augmented reality device (30) are connected by a data communication path (50).

12. The system of claim 11, wherein said sewing machine (10) and said augmented reality device (30) each comprise a transmitter-receiver means (51, 52) for establishing a wireless data transmission path (50) for data communication between the sewing machine (10) and the augmented reality device (30).

13. The system of any one of claims 1 to 12, wherein said augmented reality device (30)comprises a rechargeable battery as a power supply.

14. The system of any one of claims 1 to 12, wherein said sewing machine (10) comprises means for receiving said augmented reality device (30) in a storing position when it is not being used.

15. The system of claim 14, wherein said sewing machine (10) and said augmented reality device (30) comprise complementary contact means contacting each other and providing an electric charging connection when said augmented reality device (30) is in said storing position.

16. A method for interactive sewing operations, comprising the following steps:providing a system for interactive sewing operations comprising a sewing machine (10) and an augmented reality device (30) as defined in any one of claims 1 to 15;operating the sewing machine (10) by a user for effecting a sewing operation on at least one workpiece by using at least one thread and a plurality of operating elements of the sewing machine (10) interacting with the at least one workpiece and the at least one thread;while using the augmented reality device (30) adapted for receiving machine feedback data from the at least one sensor (20) and for presenting corresponding user feedback data to at least one of the user's senses, said augmented reality device (30) being placed between the sewing machine (10) and the user;generating, by the at least one sensor (20), machine feedback data representative of an operational state of at least one ofa) the plurality of operating elements,b) the at least one thread,c) the at least one workpiece;transmitting said machine feedback data from the at least one sensor (20, 21, 22) to the augmented reality device (30); andpresenting, by the augmented reality device (30), corresponding user feedback data to at least one of the user's senses.

17. The method of claim 16, wherein said augmented reality device (30) comprises a see-through device adapted for viewing reality information and comprising a means for displaying additional information in addition to said reality information for providing augmented reality information; whereinusing the augmented reality device (30) by the user comprises viewing the sewing machine (10) through said augmented reality device (30) together with said corresponding user feedback data.

Images